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Introduction 


The essence, of decision-aiding software is that it consists of various forms 
of microcomputer programming designed to enable users to process a set of 
goals to be achieved, alternatives available for achieving them, and relations 
between goals and alternatives in order to choose the best alternative, com¬ 
bination, allocation, or predictive decision-rule. 

Decision-aiding software should be distinguished from at least two other 
kinds of software that are relevant to making decisions but do not process 
goals, alternatives, and relations in order to arrive at prescriptive conclu¬ 
sions. One related type of software is information retrieval software. It can 
be very useful for determining such matters as the amount of money spent 
on a certain expense item in a certain year, the court cases that are relevant to 
a particular subject, or any kind of information that might be contained in a 
statistical almanac, encyclopedia, or other compendium of information. 
The second related type of software is office practice software, which can be 
useful for word processing reports, filing and retrieving in-house informa¬ 
tion, or doing bookkeeping relevant to financial matters. That kind of soft¬ 
ware is useful for organizing the decision-making processes of a government 
agency, a law firm, or any kind of office. Such software, however, does not 
process goals, alternatives, and relations to arrive at prescriptive conclu¬ 
sions. 

Decision-aiding software can take a variety of forms. The most common 
might be the following: 

1. Decision tree software for making decisions under conditions of risk, such as 

whether to go on strike or accept a management offer. A decision tree is usually 

pictured as looking like a tree on its side with branches and subbranches. The 
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branches generally represent alternative possibilities that depend on the occur¬ 
rence or nonoccurrence of probabilistic events. 

2. Linear programming software for allocating money, time, people, or other scarce 
resources to activities, places, tasks, or other objects to which the resources are to 
be allocated. In terms of form rather than function, linear programming involves 
maximizing or minimizing an objective function or algebraic equation subject to 
constraints generally in the form of inequalities like greater than or less than. 

3. Statistical software for predicting how a future event is likely to occur, such as a 
trial, an election, or a weather occurrence, in the light of past events or expert 
opinions. Statistical software generally involves calculating averages or predictive 
equations in which decisions or other outcomes are related to factual inputs. 

4. Spreadsheet-based software in which the alternatives tend to be in the rows, the 
criteria in the columns, relations in the cells, overall scores for each alternative in 
a column at the far right, and a capability for determining what it would take to 
bring a second-place or other-place alternative up to first place. 

5. Rule-based software, which contains a set of rules for dealing with a narrow or 
broad field of decision making. The user gives the computer a set of facts, and the 
computer applies the rules to the facts in order to determine which alternative 
decision should be or is likely to be decided. Such software is sometimes referred 
to as artificial intelligence (AI) or expen systems, but the other forms of decision- 
aiding software also have characteristics associated with AI and expert systems. 

6. Multicriteria decision-making (MCDM) software, which emphasizes multiple 
goals to be achieved, as contrasted to decision trees, linear programming, and 
statistical regression analysis, which emphasize a single objective function or a 
single dependent variable. 

7. Decision-aiding software that focuses on a specific subject matter, as contrasted 
to the other software, which cuts across all subjects. Subject-specific software 
could relate to how to decide where to drill an oil well, how to deal with crisis 
situations in flying a plane, or any other specific decision-making situation. 

8. Software that is useful for generating alternatives, goals, or relations but does not 
pro-cess those elements in order to draw a conclusion. 

Decision-aiding software enhances various decision-making skills. These 

include: 

1. Choosing among alternatives, where each alternative is a lump-sum choice, 
meaning that one cannot generally choose parts or multiples of such an alterna¬ 
tive. The situation can Involve mutually exclusive alternatives, or it can allow for 
combinations. 

2. Allocating scarce resources such as money, time, or people to such objects as 
places or activities. The allocating can be with or without minimum or maximum 
constraints on how much each object can receive. 

3. Explaining and predicting behavior, including Individual cases or relations, in 
either the past or the future. 

4. Teaching decision making, as well as actually making or prescribing decisions. 
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There are various obstacles to systematic decision making that decision- 

aiding software helps overcome. Those obstacles include: 

1. Multiple dimensions on multiple goals. This is sometimes referred to as the apples 
and oranges problem, although the problem appears to become more difficult if 
the goals are more abstract, like freedom and equality. The measures may simulta¬ 
neously involve hours, miles, dollars, 1-5 scales, pounds, pollution units, and 
other measures. 

2. Multiple missing information. In its simplest form, this problem involves know¬ 
ing the benefits and costs for a number of alternatives with the exception of one 
benefit or one cost. In its more challenging form, many benefits, costs, probabili¬ 
ties, and other inputs are unknown. 

3. Multiple alternatives that are too many to analyze each one separately. This is 
especially the case in allocation problems. For example, there are at least 1,001 
ways to allocate SI,000 between two budget categories A and B. One can give all 
SI,CKX) to A with SO to B, $999 to A with $1 to B, and so on down to $0 to A with 
$1,000 to B. 

4. Multiple and possibly conflicting constraints. In its simplest form, a number of 
constraints need to be met simultaneously, but they do not conflict. In its more 
challenging form, there may be minimum allocations required for each budget 
category, but the sum of the minimums adds to more than the maximum budget 
constraint. 

5. The need for simplicity in drawing and presenting conclusions in spite of all that 
multiplicity. This is where spreadsheet-based software can be especially helpful 
because it can be relatively easy to manipulate and interpret in comparison to 
decision trees, payoff matrices, systems of simultaneous equations and inequali¬ 
ties, and arrow diagrams. 

Other benefits from using decision-aiding software include: 

1. Being more explicit about goals to be achieved, alternatives available for achiev¬ 
ing them, and relations between goals and alternatives. 

2. Being stimulated to think of more goals, alternatives, and relations than one 
would otherwise be likely to do. 

3. Being able to handle multiple goals, alternatives, and relations without getting 
confused and without feeling the need to resort to a single composite goal or a 
single go/no-go alternative. 

4. Being encouraged to experiment with changes in the inputs into one’s thinking to 
see how conclusions are affected. 

5. Being better able to achieve or more than achieve goals when choosing among 
alternatives or allocating scarce resources. 

6. Being better able to predict future occurrences and explain past occurrences. 

7. Being better able to teach decision making and other related skills to students in 
courses that involve controversial issues. 
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8. Being able more effectively to handle multidimensionality, missing information, 
and multiple constraints as surmountable obstacles to systematic decision mak¬ 
ing. 

9. Being more able to deal with diverse subject matters as a result of having a cross¬ 
cutting decision analytic framework that is easy to use. 

10. Becoming more capable of systematic decision analysis, even when the software 
is not available. 

One of the most exciting developments regarding the future of decision- 
aiding software is the idea of being able to achieve superoptimum solutions, 
defined as those that are better than what each side in a controversy had 
originally proposed as its best alternative using each side’s own goals and 
their relative weights. For example, the Republicans propose retaining the 
minimum wage at S3.35 in order to stimulate business. The Democrats pro¬ 
pose raising the minimum wage to $4.00 in order to help labor. A superopti¬ 
mum solution might be to allow business firms to pay as low as $3.00 an 
hour where they agree to hire the elderly, the handicapped, mothers of pre¬ 
school children, or other unemployed people and also agree to provide on- 
the-job training. The workers, however, receive $4.50 an hour, with the 
government paving a $1.50 minimum wage supplement to the $3.00 business 
base. Business comes out ahead of its best expectations ($3.35) of being able 
to retain the existing minimum wage. Labor comes out ahead of its best ex¬ 
pectation of getting $4.00 an hour. The taxpayer is better off if unemployed 
people are put to work who might otherwise be receiving public aid, food 
stamps, Medicaid, or public housing and maybe committing crimes. They 
can now become income-receiving taxpayers. This is a superoptimum solu¬ 
tion; everybody comes out better off. It should be distinguished from a com¬ 
promise solution, which would be between $3.35 and $4.00. Superoptimum 
solutions are facilitated by thinking in terms of multiple goals and alterna¬ 
tives using spreadsheet-based decision-aiding software. 

For further information concerning decision-aiding software, see Patrick 
Humphreys and Ayleen Wisudha, Methods and Tools for Structuring and 
Analyzing Decision Problems (London: London School of Economics and 
Political Science, 1987); Saul Gass et al. (eds.). Impacts of Microcomputers 
on Operations Research (.Amsterdam: North-Holland, 1986); and S. Nagel, 
Decision-Aiding Software: Skills , Obstacles , and Applications (London: 
Macmillan, 1990). 

There are five parts to this book: 

1. Pan One provides a broad overview of decision-aiding packages, including crite¬ 
ria for evaluating them. It contains chapters by such prominent experts in the 
field as Bernd Rohraiann, Hoiger Schiitz, Charles Viek, Danielle Timmermans, 
Wilma Otten, and Ola Svenson. 
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2. Approaches that are based on management science and operations research. 
Chapter 4, by Ramesh Sharda, evaluates linear programming routines, and Chap¬ 
ter 5, by Jim McGovern, Danny Samson, and Andrew Wirth, builds on decision 
trees. 

3. Spreadsheet-based software, with insightful analyses by Steven Sonka, Michael 
Hudson, and Ron Janssen. 

4. Expert systems software, including rule-based and function-based expert systems, 
discussed by Paul J. Hoffman, and knowledge-based expert systems discussed by 
Marko Bohanec and Vladislav Rajkovic. 

5. The book ends with two chapters describing general applications and increasing 
utilization of decision-aiding software by Floyd Lewis and Fred Wood. 

The book thus provides broad coverage across a variety of methodological 
perspectives toward decision-aiding software. It also provides variety in dis¬ 
ciplinary representation, including psychology, business administration, 
medicine, law, political science, economics, and engineering. The authors 
include both academics and practitioners from Australia, Germany, Scan¬ 
dinavia, the Middle East, India, Eastern Europe, the Netherlands, and the 
United States. 




Part One 


Comparing across Software Packages 

This part provides an overview of a variety of decision-aiding software 
packages. One of its purposes is to present a typology for analyzing de¬ 
cision-aiding software. One set of classification categories is provided in 
the Contents of this book. The main categories there are (1) approaches 
based on management science and operations research, (2) spreadsheet- 
based software, and (3) expert systems software. 

The second key purpose of this part is to develop a set of criteria for 
evaluating decision-aiding software packages. Some criteria might in¬ 
clude the ability to deal in a meaningful way with such obstacles to sys¬ 
tematic decision making as multiple dimensions on multiple goals; 
multiple missing information; multiple alternatives, especially in alloca¬ 
tion problems; multiple and possibly conflicting constraints; and the 
need for simplicity in drawing and presenting conclusions in spite of all 
that multiplicity. 

On the matter of multiple dimensions, that problem gets at the idea 
that the goals to be achieved in almost any decision-making problem are 
not so likely to be measured in the same way. For example, in the sim¬ 
plest of decision-making problems, there tend to be at least some goals 
that are desired benefits and some goals that are costs to be kept down. 
It is possible to have only one goal, such as maximizing profits or net 
benefits. Traditional management science or operations research usually 
works with only a single objective function. Where “one” goal is 
present, it is usually a composite of subgoals that have been aggregated, 
frequently by forcing a common measurement unit on all the goals, such 
as dollars or another monetary unit. In a typical public policy problem 
and in most nonbusiness problems, however, the benefits tend to be non¬ 
monetary, and the costs are mainly monetary. This necessitates a 
method for determining how to subtract monetary costs from nonmone¬ 
tary benefits in order to arrive at an overall score for each alternative. 

On the matter of multiple missing information, that problem gets at 
the idea that complete facts and values tend to be missing from typical 
decision-making situations. One can arbitrarily exclude all goals or al- 
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ieraatives for which there is a lack of information concerning relations 
between alternatives and goals or concerning the relative weights of the 
goals. Doing so, however, may mean only working with relatively unim¬ 
portant goals and with alternatives that happen to be highly measurable 
or already measured. A good package is capable of dealing with prob¬ 
lems in which the user does not know the benefits and/or costs for one 
or more alternatives on one or more goals. 

“Multiple alte rnati ves’'' refers to situations in which the alternatives 
xnav be xoo many xo dete rmin e the effects of each one or even too many 
10 determine how many there are. This is especially the case in allocation 
problems. If one merely wants to allocate SI,000 to two budget catego¬ 
ries, there are 1,001 alternative ways of doing so without counting the 
pennies. One could give $1,000 to the first budget category and $0 to the 
second, $999 to the first and $1 xo the second, and so on until one gives 
$0 to the first and S 1,000 to the second. With $1,000,000 to allocate, 
there are 1,000,001 alternatives for only two budget categories. By add¬ 
ins more budget categories, the Quantity of alternatives becomes astro¬ 
nomical. No formula has yet been developed to determine how many 
alternative ways one can allocate a budget of SB to a set of K budget 
categories, although we know that the alternatives increase greatly as SB 
and X increase. A good decision-aiding package can handle such impor¬ 
tant allocation decisions as well as decisions that involve choosing 
among two, three, or more discrete manageable alternatives. 

Another criterion for judging decision-aiding software packages is 
how well they handle multiple constraints, including possibly conflicting 
constraints. The constraints could include minimum or maximum 
amounts, scores, or ratios for one or more goals or one or more alterna¬ 
tives. The constraints could also refer to restrictions on what constitutes 
an acceptable solution that are qualitative rather than quantitative. 
These constraints might be legal, political, psychological, conceptual, 
predictive, economic, administrative, constitutional, or something else. 
In a realistic decision-making problem, often there may be conflicting 
constraints, as in the common situation of having a minimum dollar 
need for food, shelter, and recreation, but the sum of those three mini- 
mums is more ihan ihe total budget available. A good package should 
be capable of handling such conflicting constraints in a meaningful way 
since good decision makers are capable of doing so without saying that 
problems containing conflicting constraints are unsolvable. 

Another important criterion for decision-aiding packages is enabling 
the user to draw and present conclusions in a simple, user-friendly way. 
Some decision-aiding software requires a knowledge of complicated 
mathematics, statistics, algebra, logic, or other quantitative techniques, 
especially the software associated with management science and opera¬ 
tions research. As a result, the software may be usable and under¬ 
standable only by those with specialized training. What is especially 
frustrating Is software that is both more complicated and less valid. That 
can frequently occur if the software requires highly precise data that are 
complicated to obtain or calculate. As a result, relevant variables may be 


Computer-Aided Decision Analysis 


3 


neglected whose absence makes the validity of the findings untrust¬ 
worthy. Good software should provide simplicity without sacrificing va¬ 
lidity, and it should especially not sacrifice both simplicity and validity 
for the sake of appearing sophisticated. 

These are all general criteria; they are applicable regardless of the sub¬ 
ject matter of the decision-aiding problem. We could develop more spe¬ 
cific criteria for certain kinds of engineering, accounting, or legal 
problems. Management science/operations research software is espe¬ 
cially applicable to industrial engineering problems. Spreadsheet soft¬ 
ware works well with accounting problems, and expert systems software 
may be especially relevant to legal problems. This book, however, is pri¬ 
marily concerned with decision analysis and decision-aiding software on 
a high level of generality. We thus tend to emphasize more general crite¬ 
ria. Not all of these criteria are emphasized in the chapters that follow, 
and some may be present implicitly rather than explicitly. Different 
people may consider some of these criteria more important than others. 
One important idea that this book seeks to communicate is that there 
are a variety of useful kinds of decision-aiding software, but beneath 
that variety there are underlying generalizations, including criteria for 
making systematic comparisons across software packages. 
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The Evaluation of Decision Aids 


Bernd Rohrmann and Holger Schutz 


INTRODUCTION: USES, USEFULNESS, AND USABILITY 
OF DECISION AIDS 

This chapter deals with the following questions: How can the quality of 
tools for aiding human decision making be evaluated, what are the pertinent 
evaluation criteria and appropriate methodological approaches, and what is 
known so far from evaluative studies of decision aids? The various existing 
concepts, strategies, techniques, and programs claim to provide a sound the¬ 
oretical basis and useful guidance for making rational decisions in the face 
of complex — usually multidimensional — choice problems. However, the 
critical question is whether such aims are actually achieved by employing 
decision-aiding technologies. To answer this question, evaluation research is 
needed. “Evaluation” means the scientific assessment of the content, pro¬ 
cesses, and outcomes of an intervention (measure, technology, program) 
and their appraisal according to defined criteria (goals, values). Only empir¬ 
ical investigation can clarify the usefulness and effectiveness of technolo¬ 
gies. 

Before discussing the evaluation problem, the interesting types of deci¬ 
sion-aiding technologies (DAT) will be specified. DAT refers to any explicit 
procedure for the generation, evaluation, and selection of alternatives 
(courses of actions, options) that is designed for practical application and 
multiple use (cf. Rohrmann 1986, p. 365). (For an overview see Humphreys 
& Wisudha 1987; Mitra 1986; Schutz & Jungermann 1988; and Vlek, Tim¬ 
mermans & Otten in this book.) Most DAT are based on decision theory as 
developed in economics, psychology, and, particularly, decision analysis 
(Bell, Raiffa & Tversky 1989; Edwards 1983; Keeney & Raiffa 1976; Keeney 
1982; Von Winterfeldt & Edwards 1986). 
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Recently, a special kind of DAT has gained importance: computerized de¬ 
cision aids or decision-aiding systems. The rapid development of computer 
technology, providing increasingly more powerful and inexpensive computer 
systems, has made the computer an easily accessible tool for decision mak¬ 
ing as well. 

There are various related computer-based tools (see Figure 1.1), including 
expert systems (ES), management information systems (MIS), and decision 
support systems (DSS), but they differ substantially in theoretical founda¬ 
tion and practical orientation. ES are usually linked to research on artificial 
intelligence. These systems consist of a knowledge base in which experts’ 
knowledge about a specific area is stored and production rules, by which the 
system can draw inferences concerning a domain-specific problem. ES may 
thus support and advise a decision maker (Fox 1984; Hayes-Roth, Waterman 
& Lenat 1983; Lehner, Probus & Donnel 1985; Sutherland 1986). MIS are 
primarily designed to structure and present dispersed data in organizations 
so that the information is easily accessible and understandable to the deci¬ 
sion maker (Keen & Scott-Morton 1978; Lee, McCosh & Migliarese 1988). 
In its narrow meaning, the notion of DSS is confined to assisting managers 
to analyze “Unstructured" decision problems (Andriole 1989; Ford 1985; 
Minch & Sanders 1986). In a broader sense, it may serve as a generic term 
for all kinds of formalized aids for decision making (and thus includes ES as 
well as MIS) in quite disparate domains, such as industrial process control, 
military tactics, commercial management, or medical diagnosis (Sprague & 
Watson 1989; Zimoiong & Rohrmann 1988); a further term, decision sup - 
pan meikods (DSM), is used by Ylek, Timmermans & Otten (this book). A 
taxonomy of -all kinds of DSS is offered by Zachary 7 (1986). Altogether, there 
is no generally accepted terminology (Seilheimer 1988; Zimoiong & Rohr¬ 
mann (1988). Only DAT deal with “decisions” in a strict sense—that is, a 
deliberate choice between options according to evaluative criteria. 

The following discussion concentrates on computerized DAT, decision- 
aiding systems (DAS) designed for personal computers and based explicitly 
on the decision-analytic approach. DAS have been designed for a multitude 

Figure 1.1 
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of purposes, including medical decision making, resource allocation, envi¬ 
ronmental conflict analysis, career counseling, and juridical and personal 
decision problems (for examples see Corner Sc Kirkwood 1991; Fraser Sc Hi- 
pel 1988; Jungermann 1980; Lehtinen Sc Smith 1985; Nagel 1988; Nagel Sc 
Mills 1991; Pauker Sc Kassirer 1981; Schiitz & Jungermann 1988; Vlek & 
Cvetkovich 1989). Most DAS, however, are not domain specific but can be 
administered to all kinds of multiattribute decision problems by providing a 
formal structure or representation to formulate the problem (Pitz 1983; Wi- 
sudha 1985). Altogether at least fifty of such DAS appear to exist. Well- 
known programs include Arborist, Decaid, Decision Analyst, Decision 
Maker, Expert Choice, Hiview, MAUD, Lightyear, Prefcalc, P/G%, and 
Supertree. Most of them are for assessing values/utilities (decision making 
under certainty); a few handle uncertainty as well (decision making under 
risk). Usually they are designed as interactive stand-alone programs for ap¬ 
plication without further support from a decision analyst or counselor. Spe¬ 
cial applications for supporting group decision making also exist but will 
not be treated here (cf. Eden Sc Radford 1990; Kraemer Sc King 1988; Phil¬ 
lips 1990). 

Three aspects of DAS can be differentiated: whether they are used (e.g., 
for what purposes, how often), whether they are useful (i.e., an effective 
tool), and whether they are usable under realistic conditions— or to put it in 
another way, usability determines to what extent the potential usefulness of 
a DAS can be exhausted. To be useful to a decision maker, a DAS must be 
applicable to her or his decision problem at hand. However, by using a DAS, 
the decision maker may be confronted with an additional problem: she or he 
has to interact with the computer (program). Thus, with regard to DAS, two 
problems can be distinguished (Streitz 1986): 

1. The content problem , dealing with the decision problem at hand to which a par¬ 
ticular analytic technique is applied. 

2. The interaction problem , resulting from the fact that the decision maker makes 
use of an interactive computer program. 

If one considers the primary reason for designing and applying DAS —to 
facilitate cognitively complex tasks —the importance of the interaction as¬ 
pect or usability becomes evident: minimizing the interaction problem 
means to set free additional (cognitive) resources of the decision maker, 
which she or he can utilize for the analysis of the decision problem itself. 
Consequently the evaluation of DAS has to take both aspects into account. 

Why is evaluation research important, and what can we learn from ex¬ 
plicit evaluation studies that will not be learned from ordinary experience? 

• Claims about the usefulness of DAS become provable in an objective and control¬ 
lable manner. 
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• Coherent thinking about the purpose and the function of the employed tool is 
enforced, and the crucial attributes of DAS are identified. 

• Empirical performance data can show whether the costs of a DAS (in terms of 
money or time or cognitive efforts) are justified. 

• If several different DAS are available, a criterion-based choice between alternative 
programs is possible. 

• The reasons that a DAS is effective or not can be explicated (because DAS are 
complex programs, the causes of success or failure are not self-evident). 

• Careful evaluation provides the best basis for systematic improvement and further 
development of DAS. 

Despite these arguments, relatively little systematic evaluation research on 
DAS has been done (Aldag & Power 1986; Rohrmann 1986). This holds even 
for DSS in general (Adeiman 1991; Adeiman, Rook & Lehner 1985; An¬ 
driole 1982; Benbasat & Nault 1990; Mahmood & Sniezek 1989; O’Keefe 
1989; Riedel & Pitz 1986; Sainfort et al. 1990; Shard a, Barr & McDonnel 
1988; Timmermans, Viek & Hendrickx 1989). 

In the following sections, first criteria for the usefulness and usability of 
DAS as well as methodological consequences for research designs will be 
discussed. Then some typical DAS evaluation studies will be reviewed. Fi¬ 
nally, some perspectives on future tasks will complete this chapter. 


QUALITY CRITERIA FOR DECISION-AIDING 
TECHNOLOGIES 

Defining evaluative criteria that are both relevant and measurable is the 
key problem for any 'evaluation study. The basic “goodness” criterion for 
technologies in general is effectiveness (Bunge 1967), defined as the degree 
to which an initial (unsatisfactory 7 or not sufficiently satisfying) situation is 
changed toward a desired state, as defined by the (normative) goal for apply¬ 
ing the technology—here, a DAT. This overall criterion has to be explicated 
and differentiated according to the specific problems and aims of users and 
the features of the employed tools. 

Evaluation criteria have been treated with respect to DAT in general 
(Riedel 1986; Riedel & Pitz 1986, Rohrmann 1986; Zimolong & Rohrmann 
1988), as well as confined to computerized DAS or DSS or ES (Adeiman, 
Rook & Lehner 1985; Mahmood & Sniezek 1989; Rouse 1984; Sage 1981). 
Table 1.1 provides a summary of pertinent criteria, reflecting the different 
perspectives of authors, designers, and users (clients, counselors, etc.) of a 
decision aid. 


Decision-Analytic Quality 

The ultimate goal of any decision maker is a good decision. However, this 
is a somewhat questionable criterion. As several authors have pointed out 


Table 1.1 

Evaluation Criteria for Decision-Aiding Technologies 


Evaluation aspect 


Evaluation competence 


Decision-analytic quality 

Theoretical/logical soundness DT 


Ability to elicit goals and preferences 

DM 

DC 

DT 


Utilization of information 


DC 

DT 


Ability to reduce judgmental biases 



DT 


Instructiveness of sensitivity analyses 

DM 




Correctness of computations 



DT 

DS 

Reliability of model results 



DT 

DS 

Congruence between problem/generated model 

DM 

DC 



Ex-post goodness of the decision 

DM 




Attitudinal effects 





Confidence in the approach 

DM 




Acceptance of procedures 

DM 

DC 



Reduction of decisional stress 

DM 




Satisfaction with results 

DM 

DC 



Frequency of application 

DM 

DC 



General and indirect benefits 





Problem clarification 

DM 




Facilitation of communication 

DM 

DC 



Improvement of decision skills 

DM 

DC 



User/system interaction 





Comprehensibility of tasks 

DM 

DC 



Simplicity of handling 

DM 



DS 

Software-ergonomic norms 

DM 



DS 

Quality/clarity of information display 

DM 



DS 

Visualization and graphical features 

DM 



DS 

Transparency of program steps 

DM 




Controllability of program course 

DM 


DT 

DS 

Possibility of input changes 




DS 

Explanatory power 

DM 


DT 


Dependency on assistance 

DM 

DC 

DT 

DS 

Quality of manual/handbook 

DM 


DT 


Flexibility 





Adaptability to tasks 

DM 

DC 

DT 


Flexibility with input data 




DS 

Adaptability to user's competence 


DC 

DT 


Usability for group situations 


DC 



Economy/Efficacy 





Time requirements 

DM 

DC 


DS 

Need for personnel 


DC 


DS 

Training necessity 

DM 

DC 



Monetary costs 

DM 




Profit 

DM 





Note: DM = decision maker (the client, end user); DC = decision counselor (or analyst, me¬ 
diator, etc.); DT = decision theorist (scientist, expert); DS = DAT software specialist. 
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(cf. Edwards et al. 1984), only the optimal usage of the information at hand 
in the decision situation should be evaluated, not the knowledge gained af¬ 
terward. Furthermore, the actual outcomes of a decision might be deter¬ 
mined by external (uncontrollable) influences rather than by the decision 
makers 7 behavior. Thus, the potential for a best possible decision should be 
increased by an aid, and decision aids should help to avoid decision traps 
(Russo & Shoemaker 1989; see also Vlek, Timmermans & Otten, in this 
book). 

The criteria listed in the first section of Table 1.1 refer to the inherent 
characteristics of a DAS. The aid—the program and the rationale behind 
it —should be sound and correct in theoretical, logical, technical, and math¬ 
ematical terms, and it should produce a valid representation of the user’s 
intentions, knowledge, and preferences. The resulting evaluation/decision 
structure must reflect and possibly clarify the cognitive problem space of the 
decision maker. 

Finally, analyzing the ex post goodness of DAS-generated decisions can be 
conclusive if the internal and external factors influencing the decision and 
its outcomes can be controlled. 


Altitudinal Effects 

The altitudinal aspects listed in Table 1.1 are relevant under two perspec¬ 
tives. Only if the user has a positive attitude toward the DAS will she or he 
be ready to learn how to make the best use of the tool and to accept and 
apply the results. Decision making under conditions of complexity, high 
stakes, uncertainty; and time pressure is stressful (Jams & Mann 1977). Re¬ 
ducing this stress is useful (even if the decision quality is not improved). 
Moreover, the subjective overall evaluation and satisfaction with a DAS can 
be measured by the frequency of its (self-intended) use. From a marketing 
perspective, any data on a product’s actual usage would be very informative. 


General and Indirect Effects 

Users satisfied with a decision aid often refer to the "insights” they gained 
(rather than to specific formal results). Accordingly, cognitive changes asso¬ 
ciated with the decision analysis should be measured by psychometric 
means. .Also, the better that decision makers understand their decision prob¬ 
lem, the easier it is to communicate with other persons involved. This is im¬ 
portant when several parties with conflicting objectives are concerned with a 
decision problem. Further, recommendations can be developed on the basis 
of a documentable decision analysis. Beyond analyzing and solving a partic¬ 
ular decision problem, a DAS can increase general decision-making compe¬ 
tence (transfer learning). 
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User/System Interaction 

The criteria discussed so far refer to the content problem of DAT evalua¬ 
tion. The fourth section of Table 1.1 deals with the interaction problem, an 
aspect that often seems to be underestimated. 

While solving a decision task with a DAS, the decision maker has to “co¬ 
operate” with a computer program —instead of working just with paper and 
pencil or discussing the problem with a decision analyst or counselor. Gener¬ 
ally, human-computer interaction has become an important issue as the 
number of users who are not trained in using computers increases. The cen¬ 
tral role of user-friendly computer systems has been recognized to be a ma¬ 
jor factor for their efficiency and acceptance (Monk 1984; Streitz 1986). 
This holds for DAS too, as several authors have noted (Bronner & De Hoog 
1983; Yon Winterfeldt & Edwards 1986; Riedel & Pitz 1986). “The challenge 
of developing an interactive decision-aiding system lies not so much in the 
design of algorithmic modules that form the basis of the system but more in 
the creation of a good interface between the user and the decision-analytic 
system. Computer technology must be challenged to its limits, while at the 
same time unnecessary complications must be avoided so as not to ignore 
the specifications for a ‘user-oriented’ system” (Wisudha 1985, p. 246). 

Such requirements have been developed in human factors research and 
particularly in the new area of software ergonomics (Shneiderman 1987; 
Streitz 1986; cf. also DeMillo et al. 1987). From a cognitive point of view, 
Streitz (1986, p. 28) has defined user orientation as the cognitive compati¬ 
bility of the user’s relevant knowledge representation with the respective 
knowledge representation of the computer system. This view applies to the 
content problem as well as to the interaction problem. Successful interaction 
with the DAS requires the development and application of an adequate men¬ 
tal model. Further, the system should minimize the cognitive load by reliev¬ 
ing the user’s memory and facilitate the perception and processing of 
information. Pertinent criteria especially relevant for DAS evaluation have 
been included in Table 1.1. 

DAS usability, however, is not just a software problem. If the instructions 
are hard to understand, if the various scaling tasks (e.g., weighting proce¬ 
dures) are difficult, or if the results are not sufficiently explained, a decision 
aid cannot be effective. 

Finally, the characteristics of the user of the system have to be considered 
(Norman 1984). Obviously, evaluation criteria are different for an experi¬ 
enced professional decision analyst, a sophisticated computer owner, and an 
occasional DAS user. 

Flexibility 

Most DAS are conceptualized as context-free tools, which, in principle, 
can deal with any particular decision problem. In reality, there are usually 
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many restrictions, depending on the characteristics of the decision task (al¬ 
ternatives, attributes, data type, etc.) and of the decision makers (laypeople 
versus experts, single versus group, etc.). Thus “adaptive” DAS (Rouse 1984) 
are needed, and flexibility becomes an important criterion. 


Economy/Efficacy 

On the one hand, the implementation of DAS produces costs in terms of 
money, time, and personnel. If, on the other hand, decision-making perfor¬ 
mance is improved by using a DAS, costs should be reduced (or even profits 
Increased). 

The evaluation criteria discussed so far are neither exhaustive nor inde¬ 
pendent, and objective measurements may often be difficult. Therefore it is 
important to consider which information source is utilized in both empirical 
and theoretical/analytical evaluations. Those expected to have the best eval¬ 
uative competence are noted on the right side of Table 1.1. Four groups are 
distinguished: 

1. Decision makers: Only they (the end users) can describe their aims, problems, 
difficulties, and so on, and only they can finally evaluate whether the employed 
tools served as an aid and led to subjective satisfaction. Also, many criteria re¬ 
quire an observation of the actual behavior of users. 

2. DAS experts: Decision scientists will have the greatest competence in theoretical 
and methodological matters, and particularly the author/designer should know 
the intended function and the potential of the DAS best. 

3. Software specialists: These experts are relevant for the evaluation of the program¬ 
ming and the user-system interfaces (including the Information provided by 
screens and printers). 

4. Decision counselors: Many DAS are employed by (or together with) a decision 
analyst who mediates between the client and the aiding system. (In some cases the 
counselor/analyst may be the virtual end user of the DAS.) Thus, these people 
have important Insights Into the usage of DAS, and they can judge whether the 
aims and needs of clients are actually met by the DAS. 

Deliberations about evaluation criteria and evaluation sources will be a 
crucial pan of designing an evaluation study. Not all criteria listed in Table 
1.1 are relevant for every 7 DAS. Instead, a critical selection of pertinent as¬ 
pects, reflecting Its specific purpose, is indicated. 

DIRECT AND INDIRECT CAUSES AND EFFECTS 

The goal of an evaluation study Is to assess whether implementation and 
use of a technology actually lead to the intended results. However, further 
causes and effects may occur (see Figure 1.2). The observed positive or nega¬ 
tive outcomes of employing a DAS can be caused by external influences 
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Figure 1.2 

Evaluation: The Causation Problem 



rather than the features of the aid. Such external factors could be, for exam¬ 
ple, the knowledge of the decision maker, the behavior of the counselor, and 
the situational circumstances. Usually, interventions have not only desired 
results but also unintended impacts. Such side effects could be confusion or 
frustration of the decision maker and reduced self-confidence, or adverse 
consequences for the relatives of a client. In addition, experimental expec¬ 
tancy biases are to be considered, regarding the subjects as well as the re¬ 
searchers. Research on the social psychology of the experiment (Adair 1973; 
Rosenthal & Rosnow 1969) has identified a number of confounding effects: 

• Subjects exposed to new working conditions respond positively when such changes 
are introduced as improvements, regardless of the particular function of the new 
conditions (so-called Hawthorne effect). 

• Experimenters have particular expectations about the effects of their treatment, 
and they may, unintentionally, convey their hypotheses to their subjects (Rosenthal 
effect). In such a case the expectations may become self-fulfilling prophecies. This 
becomes relevant if the DAS author is the evaluator. 

• Humans are likely to prefer information that supports existing beliefs: they overes¬ 
timate the value of the information in favor of a considered alternative and disre¬ 
gard opposing data (“inertia” effect). In the case of decision aids, probably both 
parties (users and researchers) have positive prehypotheses on their effectiveness 
(as already induced by the label “aid”). 

Those biases may cause a spurious pseudo-effectiveness of the analyzed 
DAS. 

In consequence, the whole picture has to be reflected in an evaluation ap¬ 
proach, and appropriate designs are to be employed in order to clarify the 
causation structure. 


DESIGNING EVALUATION RESEARCH 

When evaluating a DAT, two approaches are possible in principle: an ana¬ 
lytical assessment by experts or an empirical study that collects data from 
users (cf. Figure 1.3). Because the field of evaluation research is highly de- 
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Figure 1.3 

Types of DAS Evaluation Studies 



cross-sectional 

longitudinal 


Focus 



content-oriented 

outcome-oriented 

process-oriented 


(substantive quality) 
(summative effectiveness) 

(formative/developmental view) 


Information source 



decision maker 
decision counselor 
decision theorist 
DAS program designer 


Reference for comparisons 



unaided decisionmaking 
counselor/decision analyst 
other decision aids 
normative target values 


veloped (Cook, Leviton & Shadish 1985; Cronbacb 1982; Herman, Morris 
& Fitz-Gibbon 1988; Patton 1986; Rossi & Freeman 1985; Shadish, Cook & 
Leviton 1991; Stufflebeam & Shinkfield 1985; Wortman 1983; Wottawa & 
Thurau 1990), a variety of concepts, methods, and designs is available in 
both cases. A further relevant area from which to develop research ap¬ 
proaches is therapy outcome evaluation (Kendall & Nonon-Ford 1982; 
Smith, Glass & Miller 1986; Wittmann & Matt 1990) because there are obvi¬ 
ous similarities between (cognitive) psychotherapies and DATs (Fischhoff 
1983; Jungermann 1980). An evaluation of technologies such as DAS can be 
based on three main perspectives: 

1. Cement orientation: The analysis deals with the substantive quality of the tech¬ 
nology and its components (also called input or system evaluation). 

2. Outcome orientation: The focus is on final results and consequences of applying 
a technology (also called summative, impact, or product evaluation). 

3. Process orientation: All stages of the intervention and the development of effects 
are surveyed (also called formative or descriptive evaluation). 
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There are some basic methodological requirements (Adelman 1991; 
Riedel & Pitz 1986; Rohrmann 1986; Sage 1981): 

• Establishing criteria. The particular goals to be achieved by the DAS have to be 
stated prior to its application, and the same criteria must be measurable before 
and after the intervention (the same is true if several aids are to be compared). 
Otherwise, the evaluation of eventual effects will be questionable or impossible. 

• Specification of target population. Four kinds of subjects can be used as informa¬ 
tion sources: as “authors,” either decision theorists or DAS program designers; 
and as “users,” either decision counselors or clients (the final decision makers). 
The different expertise of these groups has already been discussed with respect to 
Table 1.1. To increase validity, all relevant groups should be included in an evalua¬ 
tion study. The same holds for different kinds of users (with respect to personal 
characteristics, substantive areas of decision making, social settings, etc.). In any 
case, the sampling decision will have critical influence on the result of the evalua¬ 
tion. 

• Study type. Most powerful but also difficult to realize (and sometimes restricted in 
terms of external validity) are experimental approaches. A wide range of quasi - 
experimental designs are available as well and of particular importance for applied 
research. Often exploratory case studies are the most suitable means. A treatment 
of these issues is beyond the scope of this chapter (cf. Bromley 1986; Cook & 
Campbell 1979; Kerlinger 1986; Miller 1991; for a discussion focusing on decision 
support systems, see also Adelman 1991). 

• Timing of data collection. Principally a pretest-posttest design is necessary for em¬ 
pirical evaluations. Especially in process evaluations, further test phases “in be¬ 
tween,” that is, a longitudinal design, will be indispensable. With respect to 
short-term versus long-term effects, also several points in time are needed. The 
timing of these phases can be crucial to distinguish between true effects and biases 
(e.g., overestimation and underestimation of the usefulness of the DAS because of 
time-dependent experiences). Furthermore, effects may change over time. 

• Inclusion of control groups. The availability of control groups may be crucial for 
interpreting the results of a study and avoiding threats to validity, such as reactivity 
effects. Principal experimental controls refer to comparisons of groups with or 
without “treatment” and with or without pretest measurement. Yet the necessary 
randomization of subjects to conditions is often difficult or impossible; the same 
is true for placebo groups or comparable nontreatment groups. In this case quasi- 
experimental research designs (cf. Cook & Campbell 1979) are indicated. 

• Selection of decision problems. The performance of a DAS can be tested only with 
respect to defined tasks. The sampling of those tasks should be representative in 
three senses: representative of the typical decision problems of clients, representa¬ 
tive of their decision-making expertise, and representative of the features and ca¬ 
pabilities of the DAS. 

• Observation of context effects. Any application of a DAS occurs in a specific con¬ 
text determined by the characteristics of the user and the social situation, by the 
decision task and its relation to other problem areas, and so forth. Therefore, the 
implementation and the course of the DAS use as well as background events 
should be carefully monitored in order to identify relevant moderator effects. 
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• Consideration of unintended outcomes. The whole set of effects of employing a 
DAS should be considered in an evaluation study. 

The design aspects strongly affect whether causal relationships can be es¬ 
tablished between the intervention (the DAS) and the outcomes that occur in 
the face of many confounding variables. Threats of internal and external 
validity necessitate systematic anticipation and prevention. (The most elabo¬ 
rated methodological position in this respect has been developed by the 
Campbell school). (For a thorough discussion related to evaluation research 
see Cook, Leviton & Shadish, 1985, or Losel & Nowack, 1987.) Choosing an 
approach should be guided by the decision-making problem underlying the 
evaluation study (Adelman 1991). Often a combination of methods might be 
most appropriate. 

The crucial stage of an evaluation study is the critical valuation of the 
observed effects. But what is the appropriate reference for comparisons? 
The outcomes of the DAS under study can be compared to: 

• Normative target values, that is, performance standards defined by the designer of 
the DAS or by the user. 

• Results of other decision aids (further DAS or similar DSS or ES) employed under 
the same conditions. 

• Decision making aided by a counselor/analyst who conducts a formal decision 
analysis (or similar noncomputerized DAT). 

• Unaided decision making, that is, a situation in which the decision maker deals 
with the task according to her or his own procedures. 

These possibilities determine the necessary design of the investigation. 

Finally, the evaluator should be independent: external evaluations will be 
more valid than those undertaken by the DAS authors themselves. In sum, 
overly simplistic approaches (or what Patton 1986 calls “quick and dirty 
evaluation”) are not appropriate in the case of DAS evaluation; instead, ad¬ 
vanced designs are required to overcome interpretative pitfalls and to gain 
reproducible, valid findings. 


KNOWLEDGE FROM EVALUATION STUDIES 

After the discussion of substantive evaluation criteria and methodological 
issues, the question arises of what is known from actual evaluation studies 
about the uses, usefulness, and usability of DAS. The number of available 
evaluation studies is still small, and these studies are heterogeneous. Thus, a 
comparative review is difficult, and certainly the body of respective research 
does not enable a formal meta-analysis (Glass, McGaw & Smith 1981; 
Hunter, Schmidt & Jackson 1982; Hunter & Schmidt 1990). Instead, se¬ 
lected examples will be described and reviewed. 
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Types of Evaluation 

Two approaches outlined in Figure 1.3 are of particular interest: compari¬ 
sons of DAS decision making with either unaided or human analyst sup¬ 
ported decision making and comparisons among several DAS. While some 
empirical investigations have been carried out on the first issue, there are 
virtually no empirical studies addressing the second one. Most studies do 
not intend to evaluate a specific DAS but address the more general question 
of whether DAS in general improve human decision making. 

In addition, a number of reviews have dealt with comparisons between 
DAS in an analytical way, providing an overview as well as a description of 
system features of available DAS, supplemented with a commentary on the 
strengths and weaknesses as judged by the authors (Henrion 1985; Radcliff 
1986; Spezzano 1985; and Taylor & Taylor 1987). Although reviews cannot 
provide an objective test, they may be useful for potential DAS users: they 
usually provide knowledge concerning theoretical soundness, utilization of 
information, simplicity of handling, monetary costs, or the kind of decision 
problem the DAS can be applied to and how it performs compared to 
others. From an evaluation perspective, these issues correspond to the evalu¬ 
ation criteria outlined. Characteristics such as theoretical soundness or hu¬ 
man engineering standards can very well be evaluated in an analytical 
fashion, without empirical investigation, and thus be treated accurately in a 
review. The major drawback is that the findings often depend on the particu¬ 
lar set of DAS treated in the review. This does not matter in regard to issues 
such as theoretical soundness or monetary costs, but it may be of impor¬ 
tance for criteria such as utilization of information or simplicity of han¬ 
dling, for which no objective or external scale of reference exists. Thus, to 
draw a general conclusion from this kind of DAS evaluation, one can say 
that DAS vary considerably in applicability, design, and sophistication of 
procedures, but one is still left with the question of their actual usefulness. 
Actual usefulness, of course, is an empirical question. 

Most empirical evaluation studies are laboratory studies, limiting the eco¬ 
logical validity of their findings. However, investigations in real settings are 
extraordinarily difficult to carry out, since it is difficult to control all rele¬ 
vant factors. In the following, the knowledge from evaluation studies will be 
summarized by focusing on three aspects: (1) Are DAS actually used, for 
what purposes, and how often? (2) Are DAS useful (effective) tools? (3) Are 
DAS usable under realistic conditions? 


Are DAS Actually Used? 

One salient operationalization of user acceptance and satisfaction is fre¬ 
quency of use. In practice, computerized decision aids are extensively ap¬ 
plied in management and organizational settings (Elam, Huber Sc Hurt 
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1986). In many cases, however, these are not DAS in the sense outlined 
above—that is, based on decision analysis -but rather MIS types of tools. 

A field for which several DAS have been designed is the area of career 
counseling. Here DAS are intended not only to support career decision mak¬ 
ing but more generally to improve the decision maker’s vocational maturity 
(Katz & Sbatkin 1983; Potocnic 1990). For example, the System of Interac¬ 
tive Guidance and Information (SIGI), designed by Katz and his associates, 
has been adopted at more than forty colleges and universities in the United 
States (Katz 1980). Wooler and his colleagues describe the development, 
testing, and use of various DAS supporting vocational decision making 
(Wooler 1982; Wooler & Lewis 1982; Wooler & Wisudha 1985). As an exam¬ 
ple, SELSTRA, a basically content-free DAS, has been specifically adapted 
to the field of career guidance by building in a core hierarchy of relevant 
attributes to stimulate the user’s reasoning -about personally important val¬ 
ues. Thai approach focuses on helping the user to construct a complete and 
realistic attribute structure. However, reports about the actual employment 
of DAS In other areas are quite sparse. For example, Jungermann, Pfister 
and May (1991) have used MAUD4 for the purpose of research on decision 
making. Bronser and De Hoog (1984) also have used a (though completely 
revised) MAUD version to conduct experiments in this area. Nagel (1988) 
gives examples of how to apply the P/G¥o computer program (Nagel & 
Long 1985) to quite different decision problems, such as deciding whether to 
get married, choosing energy policies, and comparing alternative incentives 
for reducing pollution. The latest version of P/G% comes with about three 
hundred illustrative data files, showing the DAS’s potential for a broad vari¬ 
ety of applications. The actual number and typical fields of applications 
here remain unclear, and this holds for most other DAS too. 


Are DAS Useful? 

Stiarda, Bair and Mcdonnel (1988) reviewed empirical evaluation studies 
of DSS (Including MIS) In organizational settings and found evidence for 
positive as well as negative effects of DSS on decision making. They con¬ 
clude "that field and laboratory' tests Investigating superiority of DSS over 
non-DSS decisions show Inconclusive results” (p. 144). It is not clear, how¬ 
ever, whether this Is due to methodological limitations of the studies re¬ 
viewed or to the limited effectiveness of DSS. In their own empirical 
evaluation study, the effectiveness of decision-making groups (consisting of 
ninety-six senior-level undergraduate business students) who were either 
supported or not supported by a DAS in a financial decision-making game 
was compared. Stiarda, Barr and Mcdonnel found DAS groups to make 
higher-quality decisions than non-DAS groups, as measured by the financial 
outcomes (net earnings) in the simulation. But on other quality measures, 
such as number of alternatives considered, time to reach a decision, and 
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confidence in the decision, no significant differences were found. Moreover, 
in the beginning of the simulation, DAS groups needed more time to reach 
decisions than did non-DAS groups, which may well be attributed to the 
time that subjects needed to become familiar with the DAS; later, decision 
times converged. The DAS used in the study was not a general decision- 
analytic one but rather a financial analysis tool that allows performance of 
what-if, goal-seeking, and Monte Carlo simulation analyses. Clearly, the 
findings are restricted to managerial or business decision making, since the 
subjects and decision problem as well as the DAS employed in the experi¬ 
ment are taken from this field. 

Cairo (1983) gives an overview of a number of empirical evaluation stud¬ 
ies on computer-assisted career counseling systems. He reports that the sys¬ 
tems were judged by users as understandable, helpful in facilitating career 
development, and enjoyable. Nevertheless, he concludes that “we still know 
very little about the impact that computer-assisted counseling systems have 
on the career development of users” (p. 57). The author also found some 
evidence in the reviewed evaluation studies that the systems increase users’ 
awareness of the need for planning, their concern with vocational choice, 
and their ability to relate information about themselves to potential occupa¬ 
tions. Furthermore, evidence suggests that users learn more about career 
exploration resources, acquire relevant information, consider more alterna¬ 
tives, and increase the appropriateness of their occupational preferences. 
The main measure that has been applied in the studies reported by Cairo has 
been the effect of the particular career guidance system on various indexes 
of vocational maturity, as measured with tests such as the Career Develop¬ 
ment Inventory or Harren’s Vocational Decision Making Checklist. 

A number of laboratory studies have dealt with context-free DAS applied 
to personal decision problems, rooted in classical decision theory. In an ex¬ 
perimental study, John, von Winterfeldt, and Edwards (1983) compared a 
conventional decision analysis performed by a human decision analyst with 
a decision analysis supported by MAUD3 (Multi-Attribute Utility Decom¬ 
position) (Humphreys & Wisudha 1979), a stand-alone DAS. First, thirty- 
five undergraduate college students with a decision problem currently 
important to them were identified and asked to generate at least four viable 
alternatives for their decision problem. The subjects were then divided into 
two experimental groups: the first group, consisting of twenty-four subjects, 
interacted with MAUD3 during a first session and with one of five human 
decision analysts during a second session (approximately one week later); 
the second group, consisting of the remaining eleven subjects, first inter¬ 
acted with a human analyst and then with MAUD3. Before and after each 
decision-analytic session, subjects were asked to give holistic ratings of the 
choice alternatives (they had generated before), to rank different vectors of 
alternative ratings, and to judge the usefulness of the technique (MAUD3 
versus human analyst) they had employed. Then the two sets of attributes 
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each subject bad generated in the MAUD3 and human analyst session were 
judged by three of the five human analysts with respect to completeness, 
logical as well as value independence, and overall quality of the attribute 
sets. Thus, a within-subject design was applied, allowing for comparisons of 
differences in (!) the individual preference order of alternatives resulting 
from each session; (2) the correspondence of the aided preference ordering 
with subjects’ intuitive (holistic) judgments; (3) the number and quality of 
attributes generated by the subjects in each session; and (4) the subjects sat¬ 
isfaction with the process and their confidence in the results. The authors 
found high convergence for (1) preference order of alternatives resulting 
from the~MAUD3 and analyst session (r = 0.63), as well as for (2) aided 
preference ordering with subjects’ intuitive judgments (ranging from 
r = 0.50 to r = 0.88). Further, (3) the number and quality of attributes 
generated was greater with the analyst, but the attribute sets generated with 
MAUD3 were judged to be more independent. Finally, (4) subjects’ satisfac¬ 
tion with the analyst was higher than with MAUD3, while for subjects’ con¬ 
fidence in the results, the reverse was true: MAUD3 attribute sets were 
judged to be more independent, both logically and value-wise. A final con¬ 
clusion drawn by the authors on the usefulness of DAS is that “the computer 
sessions compared quite favorably with the analyst sessions” (p. 317) and 
that “stand-alone decision aids are feasible” (p. 318), though a number of 
problems (for example, the simplification of trade-off techniques or the in¬ 
clusion of a problem-related database) remain to be solved. However, this 
study did not include a control group of unaided decision makers. Thus, no 
conclusions can be drawn about the principal benefit of systematic decision- 
aiding techniques. 

Humphreys and McFadden (1980) conducted a study in which they in¬ 
tended to clarify the question of what and how a DAS can aid in the deci¬ 
sion-making process. Provided with an (early) version of MAUD, four 
groups of subjects who were involved in decision-making problems in the 
arts and mass media first had to establish their preferences among the alter¬ 
natives during an individual MAUD session. This was followed by an inter¬ 
view’ about the established preference orderings and the users’ satisfaction 
with the process. Later, the individual judgments and preferences were dis¬ 
cussed in the respective groups. Criticizing the convergent validation crite¬ 
rion as an invalid indicator for the degree to which a DAS actually aids, the 
authors used content analysis of the interviews and group discussions to dis¬ 
cover those aspects in w^hich MAUD aided the decision makers. The results 
indicate that decision makers, whose intuitive preferences did not agree with 
MAUD-generated preferences, found MAUD most useful. Reduction of 
goal confusion and increased consciousness about the structure of value- 
wise importances of attribute dimensions were the most important effects of 
decision making with MAUD. Again, in this study no control group was 
used to compare aided with unaided decision making. In addition, the mim- 
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her of subjects (altogether twenty-two) working on a particular decision 
problem was rather small (three, five, six, and eight in each group, respec¬ 
tively). This makes it difficult to assess the generalizability of the findings. 
Also, the duration of positive effects over time was not investigated. 

Bronner and De Hoog (1983) addressed the question of how to improve 
the quality of DAS in order to facilitate intelligent and effective use of DAS 
for people without special training in decision analysis. A modified version 
of the MAUD2 computer program was used with forty subjects, who had to 
deal with two decision problems in sessions run individually. For all ses¬ 
sions, behavior observation data as well as protocols of the subject-program 
interaction were collected. Finally, subjects had to fill out a questionnaire 
concerning ease of use, perceived usefulness, and suggestions for improve¬ 
ment of the program. Most subjects judged the interaction with the DAS as 
“simple, clear and understandable” (p. 287) and found it particularly appli¬ 
cable to decision problems that were neither too trivial nor too emotionally 
important (e.g., choosing a job or an education). Similar to Humphreys and 
McFadden’s (1980) findings, helping the decision maker clarify her or his 
decision problem turned out to be a major benefit. This study focuses on 
subjective measures such as subjects’ judgments of DAS applicability and 
usability. The question of whether the use of computerized decision aids ac¬ 
tually increases decision quality is not addressed. 

Timmermans, Vlek, and Hendrickx (1989; cf. also Timmermans 1991) in 
an experimental study compared the effectiveness of decision making under 
three conditions: subjects were supported by a trained counselor or by a 
DAS or were not supported at all. The authors used two kinds of measures 
for effectiveness. Objective measures included time needed for analyzing the 
decision problem, number of attributes generated, and convergence of intui¬ 
tive preferences and preferences generated during the aided decision process. 
Subjective measures were user satisfaction with both procedure and resulting 
choice, experienced difficulty of the procedure, and acceptance of the deci¬ 
sion-analytic method. In general, the results of the study support the claim 
of DAS usefulness. Though both counselor- and computer-supported deci¬ 
sion making seems to be less useful than unaided decision making with re¬ 
gard to the time needed, more attributes were considered in the supported 
decision-making conditions, indicating a more thorough decision process. 
Moreover, subjects judged the decision-aiding procedures as especially use¬ 
ful for structuring and understanding the problem at hand. No significant 
differences were found between the computer-supported and the counselor- 
supported conditions. Compared with the unaided condition, subjects in the 
two support conditions were less satisfied with the decision process; no dif¬ 
ferences between the three conditions were found for the final choice. Fi¬ 
nally, and perhaps most important, the experienced difficulty of the 
decision problem turned out to be quite important for the effects of the deci¬ 
sion-aiding procedures. When the decision problem was regarded as diffi- 
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cult, decision support was more effective (more often led to a change in the 
final preference) than when it was regarded as simple. This indicates that 
DAS might be particularly useful for complex decision problems and that 
under these conditions decision makers are willing to rely on DAS support. 
In sum, although this study might be somewhat restricted in the subjective 
relevance of the decision problems treated, the comprehensive experimental 
approach allows for instructive and valid conclusions about the usefulness 
of DAS. 

While these experiments dealt with personal decision problems, further 
studies have addressed other aspects. Goslar, Green, and Hughes (1986), us¬ 
ing an ill-structured business decision problem. Investigated the effects of 
three factors: DAS availability, DAS training, and high and low data avail¬ 
ability on several measures of decision-making performance. Their depen¬ 
dent variables included the number of alternatives considered, the amount 
of time needed and the amount of data considered for decision making, con¬ 
fidence in the decision, each subject’s decision processing, and overall per¬ 
formance. No main effect was found on any of the dependent variables. 
However, significant interaction effects were found with regard to the num¬ 
ber of alternatives considered: subjects with DAS training, DAS availability, 
or high amount of data available considered fewer alternatives. The authors 
conclude that neither DAS availability nor DAS training enhances decision¬ 
making performance, and they make the important point that more knowl¬ 
edge on the logic of decision making itself may be required. In this 
experiment, however, subjects in the DAS availability condition interacted 
with the DAS through trained intermediaries rather than directly. This was 
done in order to exclude biases resulting from subjects’ potential previous 
computer experience. Although this measure indicates careful experiment 
planning, at the same time it poses a problem for the interpretation of the 
findings, since it remains unclear whether the failure to obtain significant 
effects is due to the fact that DAS availability was indirect and that DAS 
training was purely theoretical. In addition. Interacting with a DAS via inter¬ 
mediaries certainly is unrealistic for typical DAS usage. Finally, the decision 
aid studied was a financial analysis program and not based on the decision- 
analytic approach. 

In a similar context, Aldag and Power (1986) compared students’ com¬ 
puter-aided decision making with unaided decision-making performance 
(both judged by independent raters) in an experimental between-subjects de¬ 
sign. The authors further examined the students’ attitudes toward the DAS, 
as well as their perceptions of their decision making and performance. Over¬ 
all, attitudes toward the DAS were favorable, and limited support was re¬ 
ceived for the hypothesis that DAS users would “exhibit more confidence in, 
and satisfaction with, their decision processes and recommendations” 
(p. 576) than would non-DAS users. However DAS users were not rated to 
be superior to non-DAS users by the independent raters. The authors’ con- 
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elusion is that DAS “seem to have a high face validity and may result in 
positive user affect and enhanced confidence. To this date, however, claims 
of improved decision quality must be taken primarily on faith” (p. 586). The 
authors dealt with a rather specific set of evaluation criteria; a distinctive 
feature of the study is to employ independent raters for assessing decision 
quality. Similar results— questioning the efficiency of DSS —have been re¬ 
ported by, for example. Pots, Van Schaik, and Sol (1989) and Van Schaik 
and Sol (1990). 


Are DAS Usable under Realistic Conditions? 

When DAS are implemented within an institutional context, DAS seem to 
be quite usable. This is the case, for example, with many computerized ca¬ 
reer decision aids and with most DSS, where technical working ability as 
well as procedural advice for the user are provided. But what about the indi¬ 
vidual decision maker who has neither technical nor procedural advice at 
hand? Several studies already cited point to potential obstacles for real-life 
usability in this case. According to the distinction made in the introduction, 
two types of problems are to be distinguished: content problems and interac¬ 
tion problems. 

Although the second aspect —the criteria addressed in Table 1.1 — has not 
been investigated systematically in any of the reviewed studies, several au¬ 
thors emphasize its importance for the real-life application of DAS (Bronner 
& De Hoog 1983; Sharda et al. 1988). Yet in the light of cited reviews (Rad- 
cliff 1986) it becomes clear that not all DAS are sufficient for standards of 
“professional” software (e.g., word processors, database systems). However, 
usability—and probably even more user acceptance —of DAS depends not 
only on the soundness and theoretical quality of the decision-analytic ra¬ 
tionale and its transformation into a DAS but on the user friendliness of 
DAS as well. 

With the content problem, two aspects may be distinguished, one con¬ 
cerning the applicability of DAS to real-life decision problems, the other 
dealing with the problem of understanding and managing the procedures 
and tasks required by the DAS. The latter issue has been addressed in several 
evaluation studies. Sharda, Barr, and Mcdonnel (1988) mention the amount 
of time their subjects needed to learn how to use the DAS. In general, how¬ 
ever, the effort required to master a DAS has been neglected in DAS evalua¬ 
tion studies (although this does not generally hold for DAS evaluation in a 
business context, where “time is money”). In DAS-supported decision mak¬ 
ing, the following procedures appear to be particularly critical: the structur¬ 
ing of the decision problem (elicitation of attributes and alternatives) and 
the elicitation of attribute weights. Most DAS do not support decision prob¬ 
lem structuring, although several authors have stressed the crucial meaning 
of this phase (von Winterfeldt 1980); an exception is, for example, GHOST 
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(Landvogt, Pflsier & Jungermarin 1989). The attribute elicitation technique 
implemented in MAUD obviously poses problems for many users, as both 
Brenner and De Hoog (1983) and John, von Winterfeldt, and Edwards 
(1983) report. Moreover, these authors report about subjects’ problems with 
understanding and performing the MAUD attribute weight elicitation proce¬ 
dure, the so-called basic reference lottery tickets technique. However, recent 
versions of MAUD {Decision Analysis Unit 1986) employ a different and 
simpler weight elicitation technique, called the compensation method. In 
view of these problems. It seems quite questionable whether an inexperi¬ 
enced decision maker actually can stand alone with a so-called stand-alone 
DAS. This critique, of course, is by no means limited to MAUD. On the 
contrary, DAS that do not support these critical decision-making stages at 
all are probably less usable (and useful) for a naive user. The cited findings 
of Goslar, Green, and Hughes (1986), where DAS availability had little ef¬ 
fect on decision making for an ill-structured decision problem, support this 
conjecture. 

The applicability of DAS to real-life decisions points to a basic problem 
and applies to decision analysis In general as well. The question is whether 
those aspects of decision making typically supported by DAS are actually 
those for which people feel they need help most. This is particularly true for 
decision problems that do not easily fit. into the predetermined decision-ana¬ 
lytic problem structure, as this Is the case with many personal decision prob¬ 
lems |cf. Jungermann 1980; Jungermann & Schiitz 1992; Pitz 1987). 
'Virtually no studies address the question of whether a real-life decision 
maker with a ~reaT decision problem would actually use a DAS, even if she 
or he had a personal computer with an appropriate DAS at hand. Subjects 
of the Brenner and De Hoog (1983) study have indicated the type of deci¬ 
sion problems for which they feel computerized decision aids would be ap¬ 
propriate: decision problems at an intermediary level. But what about 
decision problems where the decision maker is involved in a highly emo¬ 
tional manner, as In genetic counseling? Pitz (1987) has given a thorough 
analysis of the difficulties of applying (noncomputerized) decision aids to 
this problem, but he also recognizes their potential benefits. Empirical evi¬ 
dence, however, is still missing. 

Taken together, only a limited number of the evaluation criteria have actu¬ 
ally been used in the evaluation studies reviewed here. Typically, measures of 
^ems.on-analytk quality, such as utilization of information, or attitudinal 
ejects, such as satisfaction with results or confidence in the approach, are 
investigated, but not as completely as suggested in the theoretical literature. 
Other evaluation criteria, such as measures of user/system interaction or of 
flexibility, have barely been taken Into account. Furthermore, the develop- 
or change of DAS effects and the stability of outcomes over time have 
been virtually ignored. In addition, methodological prerequisites for valid 
evaluations are partly missing. For example, not all evaluation studies in- 
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elude control groups. All in all, no consistent picture emerges from the sub¬ 
stantial findings. While applications of DAS generally seem to produce 
positive attitudinal effects, as yet little is known about the specific condi¬ 
tions under which DAS substantially support human decision making. 


CONCLUSIONS: PERSPECTIVES FOR FUTURE RESEARCH 

Does the available research on DAS evaluation answer the raised substan¬ 
tive questions on uses, usefulness, and usability of these tools, and are the 
discussed methodological requirements fulfilled? Obviously, this has been 
only partly achieved. In our view, both enlarged and improved evaluation is 
indicated: 

• The body of DAS evaluation studies is too small in quantitative and qualitative 
terms. For many DAS, there is no evaluative knowledge at all (only MAUD re¬ 
ceived larger interest). Also, very few evaluation results have been replicated by 
other researchers. Thus, the validity of findings is restricted. 

• There is a particular need for experimental studies in which the behavior of sub¬ 
jects is investigated by observation methods. However, such studies would reveal 
little unless strict design criteria were met. 

• The types and characteristics of decision-making problems for which each DAS 
has been evaluated are limited, and the same holds for user characteristics. How¬ 
ever, it is a well-established finding from research on behavioral decision making 
that the decision-making context is crucial for the decision process, as well as for 
decision outcomes. Thus, for many DAS, it is not clear for which kind of decision 
problems and for which type of decision makers they are appropriate or unsuited 
(see Svenson 1990, or Vlek, Timmermans & Otten, this book, for a taxonomy of 
decision problems). Further, the required competence level should be clarified. A 
comprehensive evaluation would include both novice and experienced decision 
makers and vary the complexity of tasks to be treated by the DAS. 

• A related approach would be to investigate the specific problems of users and to 
perform a kind of user needs analysis. The results could be related to tool-oriented 
evaluations. 

• Another question is whether findings can be generalized across societal groups or 
even cultures. Actually some authors stress that decision behavior is culture spe¬ 
cific (Bontempo 1990; Hofstede 1980) and deeply rooted in the philosophical tra¬ 
dition of a culture. Cross-cultural research on decision making (see McDaniels & 
Gregory 1991 for an agenda) might clarify this aspect of external validity. 

• There is virtually no empirical knowledge concerning the relative usefulness of dif¬ 
ferent DAS compared with each other. Of course, this knowledge would be quite 
valuable for potential DAS users, since the problem of choosing an appropriate 
decision aid can be a decision problem as well. Analytical knowledge, as provided 
by DAS reviews, is helpful but not sufficient. More comparative data of DAS per¬ 
formance with respect to defined types of decision tasks could aid this meta deci¬ 
sion problem. Incidentally, decision analysis is an excellent tool for evaluation 
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research (see Pitz & McKillip 1984). Thus, a DAS evaluation could easily be con¬ 
ducted via a DAS. 

• For many years the extent to which the (considerable) complexity and difficulty of 
decision-aiding technologies can be reduced without loss in precision and validity 
has been discussed. In this context, variants of DAS could be examined in system¬ 
atic exper im ents in order to analyze whether the same effectiveness and usefulness 
Is achievable with simpler procedures and less effort. 

This list of considerations and requests might sound somewhat theoretical 
or even naive. We know that proper studies in realistic settings are quite an 
effort and that usually not all reasonable goals and requirements can be 
achieved in a single study. Even worse, there might not be a great interest in 
DAS evaluation at all (perhaps outcomes would be discouraging?). Never¬ 
theless, if the aim is to aid rational decisions about decision aids, more eval¬ 
uation is indeed needed. It should be done by independent (external) 
evaluators and in a utilization-oriented way; ignored evaluations are useless. 
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The Idea of Decision Support 

Charles Vlek, Danielle Timmermans, 
and Wilma Otten 


Can we help people make better decisions? An unsuspecting Socrates at¬ 
tempting to understand this question might soon run into some troubling 
issues. First, what is a decision? Can one actually make decisions, or do they 
happen to you (and when, how)? Does decision making occur only in given 
choice situations, or may it also involve developing and structuring a prob¬ 
lem? Wouldn’t future good decision making require careful monitoring of 
the actual consequences of past and current decisions? Could the (impossi¬ 
bility of identifying current events as consequences of earlier decisions be a 
basis for measuring problem difficulty? 

Second, what are “better” decisions? Better than what should they be? Is 
the comparative baseline to be found in people’s (whose?) unaided intuition 
or in some other customary way of making decisions? Under which condi¬ 
tions can unaided decision making fairly be considered to yield baseline de¬ 
cision quality? And should the help be substantive (advising others what 
they should decide) or procedural (telling others how they should make up 
their mind)? 

Third, how could we establish that one decision is better than another? 
Should we look at the relative characteristics of the selected alternative? Are 
we to wait for the decision’s consequences and judge its quality from them? 
Or should the process of making the decision guide our quality appraisal? 
There seems to be some value in all three types of quality criteria. So when 
and for which type of problem is one or the other type of criterion appro¬ 
priate? 

Fourth, who is “people”? The individual employee contemplating chang¬ 
ing jobs, the patient facing a medical treatment, the company manager 
considering a new investment, or the policymaker selecting a site for a tech- 
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nological project? Whom are we designing decision support methods 
(DSMs) for? Or are the latter supposed to be generally applicable? 

Fifth, what involves “help” in making decisions? Self-help or others’ 
help? Personalized help or instrumented (computerized) help? Formalized 
help or heuristic help? Informational help, problem-structuring help, or op¬ 
tion evaluation help? Wouldn’t the specification of appropriate decision 
support require that one know where the difficulties, the pains, and the 
needs lie? 

Sixth, do decision makers want to be helped? It depends, undoubtedly. 
What are people’s perceptions of the quality of their own decisions? When 
do they need decision support? What triggers a call for help in decision 
making? Is it problem complexity 5 , uncertainty about outcomes, importance 
of consequences, or some other set of problem characteristics? And don’t 
we tend to link decision quality to personal conscientiousness and auton¬ 
omy? If so, wouldn’t this require that any DSM leave sufficient room for the 
user’s own view and intuitions about the decision problem, or—as Toda 
(1980) has called it —someone’s compositional rationality? 

Seventh, who is “we”? The computer specialist who develops a program 
based on a well-known decision method? The counselor who designs a gen¬ 
eral problem-solving approach based on clinical experience? The substantive 
expert who lets the data drive a decision via straightforward if-then rules? 
Or the decision scientist who is offering theoretical models for making good 
decisions? 

The design and evaluation of DSMs turns out to be the litmus test of hu¬ 
man decision theory. From a marketing perspective, one would say that 
DSMs simply have to work and, thus, to be accepted by the user. The ulti¬ 
mate user criterion, therefore, would be the belief or feeling that a DSM 
helps maxe adequate, or better, decisions. From a scientific perspective, 
however, one would need to specify which DSM is appropriate where, when, 
and for whom, and why and to what extent it will support the making of 
good decisions. Above all, the decision theorist wants to present DSMs that 
are based on a defensible concept of a “good” decision. The quality warrant 
of any DSM should be implied by its underlying model or principles or by its 
empirical basis (e.g., “experience”). Only thus can supported decision 
makers be offered external quality standards, while the DSM designer is 
freed from the subjective whims of his or her clients. 


BASIC PROPOSITIONS CONCERNING 
DECISION SUPPORT 

Let us delineate and model the concept of decision support (DS) itself 
while considering some of the questions posed above. 

First, we propose to consider decision making as encompassing three 
global stages or components: (1) acquiring, retrieving, and/or selecting 
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relevant information; (2) structuring the decision problem such that choice 
alternatives and their distinguishing features become visible; and (3) evalu¬ 
ating alternatives for their relative (expected) attractiveness. These are very 
much in line with Simon’s (1960) intelligence, design, and choice phases. We 
should realize, however, that the three components are embedded (Figure 
2.1a). Available information forms the outer context or base of decision 
making. This conditions but does not fully determine the way the problem is 
structured. The third component, option evaluation, in turn depends upon 
the other two. Figure 2.1b suggests that the three components may also be 
envisaged as forming a pyramid with three terraces decreasing in size while 
increasing in specificity. The highest and smallest terrace, involving coherent 
evaluation of given choice alternatives, is familiar ground for classical deci¬ 
sion theorists. 

Explicit DS may be addressed at all three components of Figure 2.1. 
Aimed at information acquisition, DS would imply that expert knowledge, 
from persons or documents, is provided such that the decision maker (DM) 
optimally knows the task environment in which a decision is to be made. 
When sufficient information is available, DS would involve a selection and 
ordering of decision-relevant elements such that a transparent, logical struc¬ 
ture of the problem emerges. Finally, when a well-structured problem is 
given, DS would imply a coherent evaluation of choice alternatives in the 
terms suggested by the adopted decision model. 

The general goal of DS is to canalize and direct the DM’s cognitive 
process, so as to overcome the uncertainty about what would be an accept¬ 
able or the best course of action. In principle this may involve all three com¬ 
ponents represented in Figure 2.1. 

The scheme of Figure 2.1 suggests the question of which type of DS 


Figure 2.1 

General Components of Decision Making 



Note: Envisioned as embedded ( a ) and as forming a pyramid of supporting terraces (b). 
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would generally be most effective: providing relevant information, helping 
to structure the problem, or aiding the coherent evaluation of alternatives. 
The answer may be important for understanding why clients do or do not 
accept certain forms of readily available DS. For example, a multiattribute 
utility rating technique might be unsupportive of DMs who lack basic infor¬ 
mation. More important, the issue suggests that the decision-making state 
or status of any client be carefully (self-) diagnosed before embarking on any 
kind of DS. 

Second, if DSMs are to have any practical significance, methodically sup¬ 
ported decisions should be “better” than unsupported decisions. But to be 
sensible, this simple statement must imply a ceteris paribus proviso. For ex¬ 
ample, it would be unfair to evaluate any time-consuming DSM by compar¬ 
ing it with a quick-and-dirty intuitive choice process. Hence, also, an 
evaluation criterion like the convergence of intuitive (prior) and calculated 
(final) preferences concerning a set of decision alternatives seems a tricky 
one if one is unable to specify the way in which the intuitive preferences 
came about. In fact, the quality of supported decisions cannot be properly 
established if we cannot at the same time describe the quality of comparable 
unsupported decisions, in terms of criteria like quality of basic information, 
validity of problem structure, and coherence of option evaluation. 

Third, effective DS should be directed at real clients who both need the 
help and are willing to accept it. The need for decision support may be indi¬ 
cated directly by potential clients themselves, who “can’t solve their prob¬ 
lem.” It may be more generally, and more precisely, established by studying 
the quality of unaided decisions and decision processes. Thus, the descrip¬ 
tive modeling of human decision making is essential for the design and eval¬ 
uation of prescriptive DSMs. 

Clients’ willingness to accept DS may depend on the nature of the prob¬ 
lem. As a rule of thumb one may expect that simple decisions do not need to 
be supported, wiiile overwhelmingly important problems cannot be fully 
supported. The expected success of DSMs, therefore, would seem to lie 
within the class of intermediately complex problems whose potential conse¬ 
quences do not involve strong emotional responses on the part of the DM. 

The importance of real clients implies that it is somewhat problematic to 
make use of volunteer laboratory subjects for evaluating DSMs, since these 
generally cannot be considered really to “own” the decision problems pre¬ 
sented to them. From a scientific point of view, controlled studies of DSM 
effectiveness are indispensable. The imaginary decision problems used 
should, however, be vividly representative of (the subject’s) real life. And in 
the area of DS, experimental research should always be supplemented with 
observational field studies (see Rohrmann & Schutz, this book). 

Another basic truth of successful DS is that the autonomy and the ulti¬ 
mate responsibility of the DM are always acknowledged. This again touches 
on the question of what would constitute a good decision. It seems safe to 
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assume that it is the DM who would be the ultimate judge of this. Wouldn’t 
we therefore have to accept that a good decision may depend upon several 
things simultaneously? The most obvious of these are the perceived quality 
of the selected course of action, by itself or in comparison to competing 
options, and the (perceived) quality of the way in which the decision was 
reached. Note that both criteria may comprise elements from all three global 
components of DS (Figure 2.1): information acquisition, problem structur¬ 
ing, and option evaluation. We will return to the problem of evaluating deci¬ 
sion quality. 

Fourth, it would seem that the mode of DS and the mediating agent 
should be fine-tuned toward the nature of the problem and the characteris¬ 
tics of the DM. Highly personal decisions are perhaps better supported by a 
trusted adviser or via a self-help workbook, implying only a modest use of 
formalized methods. Business investments, on the other hand, may well re¬ 
quire the use of computerized formal techniques applied to authorized input 
data. Ethical dilemmas such as often arise in health care settings may best be 
solved in a balanced expert-client interaction following an adopted decision 
procedure and subtask division. Whatever support is provided, it would 
seem that the modeling of the subject’s decision problem, based on available 
information and preceding option evaluation, is the central element in the 
process of reaching a good decision. 

The general idea of decision support unfolded thus far explicitly goes be¬ 
yond Bayesian decision theory and the methodology of decision analysis. 
Most real-life problems are sufficiently ill defined to frustrate the rational 
intentions of any DM. The principal difficulty is that Bayesian decision the¬ 
ory does not provide a stopping rule for its analysis. Hence, it seems to im¬ 
ply that the decision maker must perform an endless explication of any 
problem before he or she can justifiably cut the knot and act toward achiev¬ 
ing his or her substantive goal (Collingridge 1982). Compositional or syn¬ 
thetic, instead of analytic, rationality (Toda 1980) would often involve 
simpler or automated decision rules whose effectiveness is undisputable. 

TYPICALITIES OF UNAIDED DECISION MAKING 

Is decision support needed, and would it be accepted by real decision 
makers? This may become clear when it is known how good or bad real 
decisions are in particular situations when the DM is not explicitly sup¬ 
ported. 

Descriptive research on decision making has yielded a general picture of 
people as limited, selective, and sequential information processors. Thus, 
they are reasonably capable of making relatively simple, adaptive decisions 
(how else would they survive?). But they are poor at making complex, strate¬ 
gic decisions causing significant changes in living conditions. Let us look at 
some evidence, with a view to specifying the conditions for DS. 
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Unaided-decision quality is strongly dependent on task complexity. The 
latter mainly depends on the number of choice alternatives, of critical un¬ 
certain events, and of relevant evaluative dimensions or attributes. Complex¬ 
ity also depends on the ambiguity surrounding any decision component, 
which may especially occur when a choice alternative has long-term and/or 
widespread consequences. Novelty and perceived complexity of a decision 
problem tend to go hand in hand; growing experience works to make an 
originally complex problem look simpler. 

Limited information-processing capacity leads people to apply simplify¬ 
ing cognitive strategies to manage complex judgment and decision tasks 
(Tversky and Kahneman 1974; Tinhorn & Hogarth 1981; Hogarth 1987). 
The more difficult the decision task is, the stronger is one’s tendency toward 
simplification (Payne 1976, 1982; Payne, Braunstein & Carroll, 1978; Ol- 
shavsky 1979). Simplifying decision strategies are often effective, but they 
may also lead to systematic errors and defective choices. 

The simplification of a decision process may be achieved in a number of 
ways (Park 1978): (1) by limiting the information to be evaluated, (2) by 
screening out a number of “unacceptable” or “infeasible” alternatives, (3) by 
using noncompensatory decision rales that do not require trade-offs, and 
(4) by reducing the number of categories on the attribute scales by which 
options are to be evaluated. For example, a ten-category attractiveness scale 
may be transformed into a three-point scale yielding “positive,” “negative,” 
and “doesn’t contribute” judgments. 

Human limited information-processing capacity inevitably leads to selec¬ 
tive perception of problem elements. Salient and frequently occurring data 
axe quickly noticed, recognized, and recalled. Selective perception may 
result in an incomplete representation of a decision problem, when not all 
feasible alternatives are identified and/or not all relevant uncertainties and 
evaluative dimensions are explicated. Selective information processing may 
also yield biased probability judgments based on distorted data sets. And it 
may underlie unreliable and inconsistent value judgments due to an unstable 
structure of goals, objectives and attributes. 

Many people appear to adapt their cognitive strategies to the nature of the 
task (Brehmer 1976). Which decision strategy one employs depends on sev¬ 
eral factors, such as: personal experience, knowledge, and motivation; the 
importance of possible consequences, time pressure, and personal responsi¬ 
bility for the decision; and decision task complexity (Beach & Mitchell 1978; 
Chnstensen-Szalanski 1978). In agreement with these authors, Shugan 
(1980) contends that decision makers apply some kind of cost-benefit analy¬ 
sis whereby the time and effort required by a particular cognitive strategy are 
weighted against the importance of making an optimal decision. Under time 
pressure, people tend to use simpler and faster decision strategies; important 
decisions elicit a more elaborate approach (Malhotra 1982). 

Rational decision models seem to provide good descriptions of decision 
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behavior in simple situations. With growing task complexity, however, un¬ 
aided decision making begins to deviate systematically from the behavior 
prescribed by normative models (Payne 1976, 1982; Olshavsky 1979; Park 
1978, Malhotra 1982). Johnson (1979) compared the cognitive effort de¬ 
manded by different decision strategies applied to problems of varying com¬ 
plexity. It appeared that an increase in the number of choice alternatives did 
not enhance the cognitive demands of an elimination-by-aspects strategy 
(Tversky 1972) as much as it enhanced the effort required by an additive- 
differences strategy. 

The efficiency of various decision rules has also been investigated by 
Thorngate (1980). In a computer simulation study, Thorngate determined 
how often a particular suboptimal rule yielded the option having the highest 
expected utility (assumed linear in money) for a series of choices among 
two-outcome gambles. It appeared that cognitively simpler decision rules 
sometimes yielded suboptimal choices but often indicated the optimal or a 
close-to-optimal alternative. 

The use of simplifying strategies may lead to suboptimal decisions. 
Malhotra (1982) found that accuracy of subjects’ choices decreased with in¬ 
creasing number of alternatives and evaluative dimensions. Paquette and 
Kida (1988) also considered how subjects deal with different decision strate¬ 
gies in problems of varying complexity. For each of a number of strategies, 
they determined efficiency as a function of time required and accuracy. A 
differential decrease in accuracy with increasing task complexity was found 
for the different strategies. The most efficient DMs turned out to be those 
who applied relatively simple strategies to relatively complex problems; they 
saved time, and their accuracy hardly suffered (cf. Thorngate’s 1980 study 
discussed above). With increasing task complexity, the accuracy of choices 
under an additive compensatory decision rule decreased more strongly than 
under the less demanding elimination-by-aspects rule. 

A significant cause of deficient decisions is stress due to sheer problem 
complexity, information overload, importance of consequences (risk), lack 
of hope for an acceptable course of action, shortage of time, and social 
pressure. The associated deficiencies are, respectively: simplifying decision 
strategies, selective use and neglect of information, undue weighting of neg¬ 
ative aspects of consequences (risk avoidance), a defensive instead of func¬ 
tional attitude toward the problem, hypervigilant or panicky information 
acquisition, and unjustified conformity to others’ opinions or enclosure in 
groupthink (Janis 1972). Janis and Mann (1977) have noted that stress — 
from whatever source—generally precludes what they call “vigilant informa¬ 
tion processing,” which they equate to efficient decision making. 

There are some other phenomena that restrict the quality of human deci¬ 
sions. Premature and rigid commitment to a chosen course of action would 
go against the mental openness that may be required for a needed course 
correction on the way (Staw 1981). Related to this, but more serious in de- 
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gree, is the power of addiction, which at an emotional-physiological level 
may pull a DM back into doing something that he or she does not want to do 
(any longer) at a conscious level (Sjoberg 1980). Finally, a natural peculiarity 
of human decision making is the discounting of future events, positive or 
negative. This explains, for example, why we do not easily pay immediate 
costs to secure future benefits and why we tend to be quick in reaping the 
fruits of today despite the associated long-term costs and risks (Bjorkman 
1984). 

In view of the nature of unaided decision making, one would say that 
explicit DS is to serve various functions. Especially for novel and complex 
decision problems, DSMs should help overcome the implications of cogni¬ 
tive limitations such as selective perception and retrieval of information, in¬ 
complete or biased definition of the problem, and unreliable or inconsistent 
evaluation of choice alternatives. The power of DS is that it extends the 
DM’s informational, reasoning, and evaluative capabilities so as almost to 
make him or her operate as a simultaneous instead of a sequential informa¬ 
tion processor. Conversely, however, the power of experienced unaided DMs 
lies in the relative efficiency of simplifying strategies. Thus, it would seem 
that any recommendable DSM should not only result in optimal, or accept¬ 
able, decisions. It should also be efficient in the context in which the deci¬ 
sion problem occurs. 

PROCESS COMPONENTS AND A PROBLEM TAXONOMY 

The easiest distinction is between simple and complex decision problems 
or between well-defined and ill-defined problems. The suggestion was made 
that novelty of a problem correlates with its perceived complexity and lack 
of definition. For DS to be effective, it would be crucial to know why and in 
what respects a decision problem may be complex or ill defined. This knowl¬ 
edge would enable us to seek an adequate decision method and rule, the 
proper criteria, and the necessary information, and it might alert us to cer¬ 
tain errors of judgment that would be likely to occur. 

A general definition of a decision problem is as follows: 

A decision problem occurs when a decision maker: (a) notices a discrepancy between 
an existing state and a desired state, (b) has the motivation as well as (c) the potential 
to reduce this discrepancy, whereby (d) there is more than one possible course of 
action which may not be immediately available, (e) the implementation of a course 
of action demands an irreversible allocation of his resources, and (f) the utilities (of 
the consequences) associated with each choice alternative are partly or entirely un¬ 
certain. (Vlek & Wagenaar 1979, p. 257; adapted from MacCrimmon 1972). 

This definition in principle applies to the worst-defined decision problem 
anyone might encounter. It would therefore be useful to sketch a functional 
decision process aimed at attacking (very) ill-defined problems. This is done 
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in Table 2.1, which reflects the problem-structuring and option-evaluation 
components of the general delineation of decision support given in Figure 
2.1. Note that the option-evaluation part of the process is formulated in 
terms of Bayesian decision theory; it might well be filled in otherwise. 

In terms of the (prescriptive) process model of Table 2.1, decision prob¬ 
lems may be more or less complex or difficult to handle. This partly depends 
on task and situational characteristics and partly on personal resources and 
peculiarities; no problem can be defined independently from its owner. Let 
us consider ten problem variables — not all mutually independent —that may 
serve to categorize decision problems. These variables can be grouped ac¬ 
cording to whether they concern problem structuring, option evaluation, 
and the (social) context of the decision problem proper. 

Variations in Problem Structuring 

1. Availability of choice alternatives. A possible course of action may be imagina¬ 
ble, but it may not (yet) be feasible. Alternatives may also become only sequen¬ 
tially instead of simultaneously available. 

2. Ease of consequence description. Easy consequences may not be a problem; dif¬ 
ficult consequences may call for special studies. 

3. Short versus long time horizon. Relevant consequences may be fairly immediate 
or remote in time. The latter are often highly uncertain, and they pose special 
problems for their evaluation. 

Variations in Option Evaluation 

4. Single-atiributive versus multiattributive consequences. Utility assessment for 
these different types of consequences is quite different, and different techniques 
are available. 

Table 2.1 

Successive Components of a “Conscientious” Decision Process 

I. PROBLEM STRUCTURING a. problem identification: establishing discrepancy 

between existing and desired states; goal 
analysis 

b. generation/search/design of feasible choice 
alternatives 

c. description of alternatives in terms of 
uncertain events, contingent choices and 
eventual consequences 


II. OPTION EVALUATION d. assessment of probabilities of uncertain 

outcomes and (optional) acquisition and 
processing of new information 

e. utility assessment in view of adopted goals/ob¬ 
jectives; trade-offs among cost, risk and 
benefit dimensions 

f. overall evaluation and rank ordering of 

alternatives; (optional) revision and re¬ 

ordering after sensitivity analyses 
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5. Riskless versus uncertain or risky choice alternatives . This is a classic taxonomic 
variable associated with fundamentally different decision models and with the 
specific subtask of probability and risk assessment. 

6. Internal versus external uncertainties . This variable is nested under the category 
of risky decision problems. Internal (controllable) uncertainties depend upon 
the DM*s skills and knowledge concerning a course of action. External (uncon¬ 
trollable) uncertainties depend upon factors outside the DM’s control, some¬ 
times referred to as acts of God. 

7. Frequemistk versus nonfrequentistic event information . This equally nested 
variable pertains to the availability of frequentistic experience (or statistics) 
about the outcomes of an uncertain event. In novel decision situations, experi¬ 
ence is lacking and one has to assess probabilities by modeling a stochastic 

process or by constructing ad hoc scenarios. 

In combination, varieties 6 and 7 define at least four rather different kinds of 
uncertainty (Kahneman & Tversky 1982; Hendrickx, Vlek & Oppewal 1989), which 

may underlie probability estimates (Figure 2.2b). 

Variations in (Social) Context 

8. Degree of personal freedom of choice. The DM may be more or less forced to 
select a particular course of action. This may be due to social pressure, to the 
sheer necessity' of fulfilling basic needs, or to his or her being tied to an unde¬ 
sired habit or addiction. Freedom of choice brings with it personal responsibili¬ 
ties, and “responsible” DMs may need specific support. 

9. Number of involved persons. This variable may overlap with variable 8: many 
others involved may mean little personal freedom. Variable 9, however, is pri¬ 
marily meant to indicate the number of persons involved in the possible conse¬ 
quences of a decision: will the costs, risks, and benefits be borne by the DM 
alone or by many others as well? In the latter case, the evaluation of conse¬ 
quences invites special considerations and techniques. 

10. Reversibility of decision or restorability of consequences. Although the general 
definition says that a decision involves an irreversible allocation of resources, the 
possibility of reversing a decision and/or restoring its consequences when they 
occur significantly influences the importance of generating feasible courses of 
action and making a good decision. Irreversible decisions having unrestorable 
consequences amount to the purest of gambling situations. No DM likes this 
very much; most people cherish control and flexibility, before and after the die is 
cast. 

The ten taxonomic variables and their division in three groups are summa¬ 
rized in the triptych of Figure 2.2. Here, all 2 10 = 1,024 theoretically possi¬ 
ble types of decision problem have been conveniently reduced to just 26, 
labeled with the letters of the alphabet. Obviously, variations within the 
other two panels of Figure 2.2 should be considered once a decision problem 
has been characterized within a given panel. 

The taxonomy suggested in Figure 2.2 does not explicitly address the in- 


Figure 2.2 

Taxonomy of Decision Problems 
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formation acquisition component of DS (cf. Figure 2.1). This is because in¬ 
formation acquisition is more often triggered by the recognition of a 
decision problem than that it leads into the problem by itself. Thus, the 
search for relevant information may unfold and intensify as one goes 
along the process components of Table 2.1. The nature of the informa¬ 
tion required depends upon the type of decision problem, as suggested by 

Figure 2.2. . . 

The proposed taxonomy also does not seem to incorporate the classical 
distinction between static and dynamic decision problems (Edwards 1961; 
Toda 1962; Rapoport 1975). On the one hand, however, dynamism is repre¬ 
sented in Figure 2.2—for example, if one considers a decision problem with 
choice alternatives not (all) immediately available (variable 1), a long time 
horizon (variable 3), internal and nonfrequentistic uncertainties (variables 6 
and 7), and a reversible choice or restorable consequences (variable 10). On 
the other hand, a dynamic decision task is very hard to model formally and 
to support methodically (cf. Klieter 1975). Perhaps classical decision theory 
is not suitable for such problems, and some other perspective (such as deci¬ 
sional control theory) is required. 

Figure 2.2 shows some resemblance to a detailed taxonomy for character¬ 
izing risk-decision problems, mapped out by Merkhofer (1987, pp. 50-53) 
in forty-four dimensions grouped into ten considerations: complexity of al¬ 
ternatives, complexity of relationships, nature of relevant information, out¬ 
come valuation complexity, constraints, decision-maker characteristics, 
organizational characteristics, sensitivity of public perceptions, controversy, 
and expectations. Evidently, the technological risk management problems 
considered by Merkhofer occur in a multiple-actor context of different kinds 
of information, complex action-consequence relationships, and possible re¬ 
actions of an affected public. Similarly as for Figure 2.2, the challenge for 
decision supporters would be to map Merkhofer’s (1987) problem dimen¬ 
sions into distinct decision approaches and methods. 

The methodological implications of Figure 2.2 are numerous and diverse. 
For example, not immediately available alternatives (variable 1) must be 
searched or designed, and this may require special methods or strategies. 
Sequentially presented alternatives necessitate the formation of an accept/ 
reject criterion (possibly multiattributive) based on the evaluation of an ini¬ 
tial sample (subsequence) of candidates. For the utility assessment of 
consequences expected in different time periods (variable 3), so-called time 
preferences may have to be modeled. Methods for the assessment of proba¬ 
bilities (variables 6 and 7) depend on the kind of information available. The 
possibility of irreparable negative consequences (variable 10) puts special 
demands on the assessment of (dis)utilities, and it may require a specific 
decision rale or principle. In general, a specific type of decision problem in 
the framework of Figure 2.2 invites the use of a specific set of concepts and 
methods, often with assumptions of their own. Similarly, different cognitive 
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shortcomings and peculiarities may variously manifest themselves in con¬ 
nection with the varying types of decision problem. Decision supporters 
may want to beware of them. 

Figure 2.2 should be a useful taxonomy insofar as it allows for a mapping 
of different types of decision problem into different decision models and/or 
different composites of decision methods and techniques. This taxonomy 
somehow Incorporates or further specifies classical as well as more recent 
distinctions made by, among others, Edwards (1961), von Winterfeldt and 
Fischer (1975), Keeney and Raiffa (1976), Vlek and Wagenaar (1979), Von 
Winterfeldt (1980), and Larichev (1984). 

Decision theorists may realize that Figure 2.2 is just too much (for what 
methods we have to offer), that there are strong ecological correlations 
among the ten variables (so that addition of a few more would not really 
matter), and that a more global and much simpler categorization of decision 
problems, models, and methods would be sufficient. This may be accepted 
for the moment, and we will consider such a strongly reduced division in 
terms of different levels of cognitive processing —or modes of decision mak¬ 
ing, evoked by strategic, tactical, and operational decision problems, respec¬ 
tively. 

LEVELS AND MODES OF DECISION MAKING 

Rational decision analysis, the methodology developed from Bayesian de¬ 
cision theory (Raiffa 1968; Schlaifer 1969), has been elaborated in detail; 
von Winterfeldt and Edwards (1986), for example, discuss the interplay be¬ 
tween analytical tools and cognitive processes. But decision analysis has not 
caught on as much as its originators had expected because it is too analytical 
for an effective approach to most practical decision problems. It also implies 
too static a conception of what a decision problem actually is. 

Based on the ecological correlations among the ten taxonomic variables 
(and with apologies to the empiricists), we may consider judgment and deci¬ 
sion making as occurring at three different levels of cognitive processing: the 
strategic, the tactical, and the operational. This is a pragmatic division that 
enables us to link major types of decision problems to different models and 
criteria for decision making. 

Roughly, an operational (or “easy”) problem is characterized by directly 
available choice alternatives, easy-to-assess consequences, a short time hori¬ 
zon, single-attributive consequences, no or only few uncertain events, inter¬ 
nal (controllable) and frequentistic uncertainties (if present), a large 
freedom of choice, few other people involved, and a reversible choice and/or 
restorable consequences. Operational choice problems occur frequently, and 
they tend to be repetitive, so that (statistical) experience may be accumu¬ 
lated. They also tend to form a dynamic chain of (control) actions, together 
aimed at some greater-than-local goal. 
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Strategic (or “difficult”) decision problems, on the other hand, are quali¬ 
fied by the opposite poles of the ten taxonomic variables: alternatives are 
not immediately clear, nor are the consequences; the latter are long term and 
multiattributive in nature; there are various uncertain events due to (uncon¬ 
trollable) external factors with which there is little experience; there is lim¬ 
ited freedom of choice; many others are involved; and an eventual decision 
would be irreversible while consequences could not be restored. Strategic 
choice problems tend to be novel or unique; frequentistic experience about 
them is lacking. Because of the external uncertainties involved and the long 
time horizon of relevant consequences, strategic decisions often amount to 
mammoth gambles. 

Tactical decision problems are intermediately characterized between oper¬ 
ational and strategic ones. In fact, operational, tactical, and strategic deci¬ 
sions form an embedding structure. Strategic choices determine the 
conditions under which tactical choices must be made. The latter, in turn, 
set the scene for operational choices, which are mostly of an adaptive nature 
(which is why we are good at them). The paradox of Bayesian decision the¬ 
ory is that it cannot be validly applied to unique or “experience-free” prob¬ 
lems, whereas it is not dynamic enough to apply to the many chained 
operational problems making up much of daily life. 

Operational, tactical, and strategic decisions are made at different levels 
of cognitive processing; they follow different modes of decision making; 
and they require different kinds of decision support. Operational choices are 
often habitual or automatic, following fixed if-then rules (“production 
rules”; cf. Anderson 1983). The programming of such rules gradually takes 
place as experience accumulates. The action-oriented routines underlying 
operational choices are highly effective, and disturbing the routine may lead 
to ineffective behavior. For the behavioral routine to remain in operation, an 
operational choice should generally have rewarding consequences. 

Tactical choices are halfway between routine and unique. It would be too 
risky to make them automatically, but they are not serious enough to war¬ 
rant an in-depth analysis. The proper way to make tactical decisions is by 
(sequentially) testing available courses of action against a set of acceptabil¬ 
ity criteria. This is traditionally called “satisficing” (Simon 1957), which may 
be done under different formal models. 

Strategic decisions are the most complex and crucial ones. They therefore 
invite the application of a full-fledged decision analysis; the decision rule is 
to seek the optimal course of action, which has the highest expected utility. 
This is customarily labeled maximizing. 

Thus, we have identified three levels and modes of decision making, each 
of them associated to a particular overall decision criterion. This is summa¬ 
rized in Table 2.2. The inference Table 2.2 allows us to make is that choices 
and decisions (philosophers may discriminate among them) may be made 
sooner and more automatically or later and more analytically. Nonanalytic 
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Table 2.2 

Three General Types of Decision Problems 


type of problem 

generic approach 

nature of choice 

strategic 

analyzing 

’optimal * 

tactical 

standardizing 

* acceptable * 

operational 

automatizing 

* rewarding' 


Sore; Most of these empirical criteria can also be used to assess the quality and effectiveness of 
a subject’s own (unaided) way of decision making, which we study as a control condition 
with the aid of think-aloud protocols. 


choices may not be bad at all, nor are analytically taken decisions always 
good. Each of the three ways of decision making —by routine, through satis¬ 
ficing, or via maximizing —has its own quality warrant-the strength of 
expected reward, the satisfying of explicit acceptance criteria, and the maxi¬ 
mum expected utility, respectively. 

The pragmatic three-level distinction proposed here is compatible with 
systematic distinctions made by others. Simon (1960) distinguishes between 
programmed and “open” (ill-structured) decision problems. One can con¬ 
sider a continuum of judgment and decision-making tasks, ranging from 
“intuitive” to “analytical.” Fischhoff and associates (1981) systematically 
compare three different approaches for evaluating the acceptability of risks: 
expert judgment, bootstrapping the decision maker, and formal analysis. 
These would seem to run reasonably parallel to the (expert) automatizing, 
(explicit) standardizing, and (comparative) analyzing modes proposed in 
Table 2.2. 

Svenson (this book) usefully separates four levels where decisions are 
reached by: (1) matching alternatives against an ideal or reference option, 
(2) applying some noncompensatory rule, (3) using a compensatory rule re¬ 
quiring -trade-offs, and (4) an elaborate problem structuring, alternative- 
generation and maximization process. Humphreys and Berkeley’s (1984) 
five-level system for analyzing decision problems seems to imply a decision- 
analytic Einstellung; it may be especially useful within the class of strategic 
choice problems, where analysis may be deeper and broader or shallower 
and narrower, depending on the extent to which the problem is already de¬ 
fined. We would suggest that their taxonomy parallels the three-component 
scheme of decision support given in Figure 2.1: information acquisition 
(their level 5), problem structuring (their levels 3 and 4), and option evalua¬ 
tion (their levels 1 and 2), respectively. 

Taxonomies of decision making may be designed in terms of problem 
characteristics (as in Figure 2.2), in terms of process components (as in Table 
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2.1 and, more globally, in Figure 2.1), in terms of decision rules or criteria 
(as proposed by Svenson, this book), in terms of levels of abstraction or 
generality (or width of scope; as done by Humphreys & Berkeley 1985), and 
in terms of mode or type of decision making (as in Table 2.2), and as sug¬ 
gested by Hammond et al. (1980). To the extent that the term decision mak¬ 
ing has a generally accepted meaning, taxonomies pertaining to decision 
making should be interrelated. Table 2.3 demonstrates this (suggestively) for 
the taxonomies discussed. 

In general, the more strategic a decision problem is, the broader and 
deeper analysis it provokes, the more constructive the decision process is, 
and the more maximizing the decision maker’s overall strategy tends to be. 
Since this would require intense cognitive efforts and other resources, it is 
fortunate (and no accident, from an evolutionary point of view) that strate¬ 
gic decision problems are greatly outnumbered by tactical and operational 
problems, where satisfactory and rewarding choices are made following 
standardized and automated rules. We will survey formal models for stand¬ 
ardized and analytic decision making, respectively. Formal models for au¬ 
tomatized decision making belong to the realm of psychological learning 
theory. Although learning-theoretic principles may usefully serve to design a 
decision-making expert system, their treatment falls outside the scope of this 
chapter. 


WHAT CONSTITUTES A “GOOD” DECISION? 

Theoretically the quality of decisions may be evaluated in two different 
ways. The first is to consider the value of the average outcome of a series of 
comparable decisions (each separate decision’s outcome may be codeter- 
mined by good or bad luck). Because of the requirement that decision out¬ 
comes be easily identifiable, this quality criterion is best applicable to 
frequently occurring, relatively simple decisions that may be standardized. 
Note that a direct-outcome criterion would naturally fit the simplest riskless 
choices, which may be automated: “a good choice should (simply) have a 
rewarding outcome.” Outcome criteria relate to a DM’s substantive rational¬ 
ity. 

The second way is to evaluate a decision by its procedural rationality. This 
is especially Important when it is difficult or impossible to trace a decision’s 
actual consequences, either because they occur too far into the future or 
because they may also be attributed to other decisions. This is usually the 
case with relatively rare and complex decisions that elicit analysis. 

Decision rationality is not easy to define. According to Toda (1980, 
p. 139): “In my opinion, ‘rational’ is an adjective meaningfully applicable 
only to the principle or rules under which a purposive system operates. A 
principle or rule is rational if and only if it makes the system function (al¬ 
most) optimally for a given context and under given constraints.” Four kinds 
of rationality may be usefully distinguished: 


Table 2.3 

Suggested Parallelisms among Different Taxonomies of Decision Making 

type of decision analytic level of mode of cognitive" 'FormnIdocinion focu7of*decioion 

problem abstraction processing strategy process 

(H. k B. 1985) (Svenson, this vo! 
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1. Representational rationality: Knowing the decision task environment. 

2. Goal-value rationality: Knowing one’s goals, objectives, and values. 

3. Methodical rationality: Coherent assessment of decision inputs and use of deci¬ 
sion rule. 

4. (De)compositional rationality: Striking an adequate balance between intuitive 
and analytical judgment. 


The first three can be maximized; one may always try to improve one’s 
knowledge of the relevant “small world” and the elaboration and consistency 
of one’s value system, and to refine and strengthen the assessment of inputs 
for the decision process. The fourth kind of rationality, however, somehow 
spoils the game. Neither an overanalytic nor a too superficial consideration 
of one’s problem yields the “best” decision. Under the principle of (de)com- 
positional rationality, a balance is required between intuitive synthesis, 
which may cut (or keep) the process short, and prolonged analysis, by which 
it may be paralyzed. 

Note that both representational and goal-value rationality reflect substan¬ 
tive rationality: one knows the (small) world and one’s own values, and this 
is the best warrant for obtaining good decision outcomes. In contrast, me¬ 
thodical and (d)compositional rationality are reflections of procedural ra¬ 
tionality. Given what one knows and does not know, the assessment 
methods, depth of analysis, and decision rule followed guarantee that an 
optimal decision is made. 

Rationality works out differently for different types of problems. For rela¬ 
tively simple problems that may be standardized, representational and goal- 
value rationality have somehow been filled in already (in the past); the 
problem is well known, and one largely knows what one prefers. This is why 
methodical rationality may be laid down in fixed rules for judgment and 
decision making. Thus, the fourth kind of rationality is strongly composi¬ 
tional or synthetic. 

For complex, strategic problems, however, methodical rationality should 
be dominant since it fulfills the crucial function of bringing representational 
and goal-value rationality up to the required level. In other words, one tries 
to structure and evaluate the decision problem as coherently as possible, and 
this demands considerable analytic or decompositional efforts (the fourth 
kind of rationality). 

Within classical decision theory, the concept of rationality has been 
operationalized in terms of a handful of axioms applicable to well-defined 
decision problems (see Krantz et al. 1971; also listed in Vlek & Wagenaar 
1979, p. 274). For example, decidability implies that a DM for any pair of 
choice alternatives, A and B, can indicate that he or she either prefers A or B 
or is indifferent between the two. Transitivity means that the preference pat¬ 
tern for any three or more options is not cyclic: when A is preferred to B and 
B to C, then A is also preferred to C (instead of C to A). Dominance in- 
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volves that a DM facing two options always prefers the option evaluated as 
at least as good as the other in all respects and as better than the other in at 
least one respect. The sure-thing principle states that the preference relation 
among two options having uncertain consequences is independent of the 
value of a fixed consequence (“sure thing”), which follows the occurrence of 
a particular outcome of an uncertain event, no matter what one chooses. 

Formal decision models and methods differ in the degree to which they 
are tuned toward satisfactory or toward optimal choices. Satisficing as an 
overall decision strategy involves a choice in favor of an “acceptable” option 
that is considered “good enough” (for the moment): this notion reflects the 
importance of compositional, standardized rationality. An optimizing deci¬ 
sion strategy is aimed at selecting the best possible course of action; it is an 
operationalization of decompositional, problem-specific rationality. Formal 
models of decision making for problems that have already been defined are 
surveyed in the next section. 

So far it would appear that decisions are “good” to the extent that they are 
rational (which means something different for simple and for complex deci¬ 
sions). Thus, a particular DSM might be evaluated as good to the extent that 
it helps one make rational decisions. However, what would make you believe 
that a given DSM would lead you to a rational decision, and would the 
DSM’s inherent rationality be compatible with the rationality you see fit for 
your decision? And do you think that your actual problem can be structured 
and evaluated rationally at all? In other words, shouldn’t we also consider 
more pragmatic and specific criteria for evaluating the quality of decisions 
and of DSMs? Yes, we should, and Rohrmann and Schutz’s contribution to 
this book (Chapter 1) testifies to this. 

In our own empirical research (Timmermans, Vlek & Hendrickx 1989) we 
evaluated computer programs for decision support (De Hoog’s 1983 MI¬ 
DAS; Pitz’s 1987 DECAID; and Sonnenberg & Pauker’s 1987 DECISION 
MAKER; see also Lau, Kassirer & Pauker 1983) by relying on four main sets 
of variables: (1) process characteristics such as time needed (per compo¬ 
nent), number of attributes used, and number of changes after sensitivity 
analysis; (2) robustness of the final choice made, assessed in terms of the 
subject’s trust and acceptance, possibly against post hoc criticisms by 
others; (3) changes in the subject’s view of the decision problem, such as 
through deeper understanding; and (4) personal satisfaction with the deci¬ 
sion method followed, with specific comments on separate components 
(e.g., “I find the assignment of weights the most burdening subtask”); here, 
the personal autonomy “respected” by the DSM may play an important role. 

Naturally, these empirical assessments come on top of the DSM’s inherent 
rationality characteristics that mainly depend on the underlying decision 
model. And thus the concept of decision quality or DSM effectiveness ap¬ 
pears to be a multidimensional one, which suggests that one might well uti¬ 
lize multiattribute utility models to support one’s comparative evaluation of 
decision aids (see also Reidel & Pitz 1986 and Pitz 1987, who extensively 
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discusses the evaluation of decision aids). Ozernoy (1988) considers the se¬ 
lection of an appropriate multiattribute decision method under a limited hi¬ 
erarchy of objectives: (1) mutual correspondence between the method and 
the decision problem, (2) validity of the method's assumptions regarding 
DM preferences and compatibility with preference information, and (3) re¬ 
source demands (e.g., costs, time) of the method. 

FORMAL MODELS OF DECISION MAKING 

In order to delineate and operationalize the concept of decision support, 
we have thus far taken a broad view resulting In various taxonomic consider¬ 
ations and a circumscription of what constitutes a “good” decision. In this 
section we narrow down to the formal models that provide practical tools for 
decision support: satisficing models for “standardized” decision making and 
Bayesian decision analysis for “maximizing” choices. 


Satisficing Models: Utilizing Acceptance Rules 

This class of models provides operationalizations of more or less compli¬ 
cated if-then rules. An example is: “If the annual costs are low and the bike 
is a reliable mode of transport, if the distance isn’t too large, the trip itself 
not too strenuous and the weather is fair often enough: then I judge regular 
bicycling to and from my office to be an acceptable course of action.” 

Such testing methods are applicable only when several conditions are ful¬ 
filled. First, obviously one or more choice alternatives should be available. 
Second, the DM should be able to evaluate these in terms of a set of measur¬ 
able variables. Third, a value or utility function must be defined on each 
variable such that acceptability judgments or trade-offs against other varia¬ 
bles can be made. Fourth, when there is more than one variable, a combina¬ 
tion rale must be adopted that indicates how the judgments on the separate 
variables are combined into an overall statement: accept or reject this op¬ 
tion. Fifth, when one or more options appear to be judged acceptable, a 
supplementary decision rale is needed in order to make a singular choice. 

Methods for testing options against acceptance criteria are thus aimed at 
(only) satisfying the DM. In principle, the decision process ends as soon as 
an acceptable option has been identified. Only when alternatives are easily 
available (generated, presented, or designed) may one attempt to select the 
“best” option from the subset of acceptable ones. Therefore, a sequential 
presentation or confrontation of alternatives would be sufficient for a satis¬ 
ficing model to apply. At every instance one actually has three options: 
(1) accept this alternative, (2) reject it and continue searching for an accept¬ 
able one, and (3) reject it and adhere to the status quo (do nothing further). 

Testing choice alternatives one by one against a set of acceptance criteria 
can be a very efficient and time-saving decision method. Crucial questions. 
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however, are: is the DM using clear and valid enough acceptance variables; 
does reliance on the adopted criteria indeed result in good decisions; and are 
the chosen criteria jointly sufficient for making singular choices? 

Testing options against acceptance criteria amounts to following a stan¬ 
dardized decision rule. The selection of relevant variables and the adoption 
of a combination rule, therefore, involve a set of prior (meta-) decisions by 
which the rule is, or ought to be, calibrated to the (statistical) database that 
lends it its validity. Revision of the standards-testing rule may be regularly 
needed as time goes by and experience with additional cases accumulates. 
Thus, the good decision maker here exploits his or her knowledge of the 
(small) world and of his or her own positive and negative experiences (i.e., 
his or her substantive rationality). This again demonstrates that standards- 
testing models of decision making are best suited for relatively simple,, fre¬ 
quently occurring decisions that allow a DM to evaluate their quality by 
considering actual decision consequences —on the average, that is. 

There are several different satisficing models and rules. Suppose that x u 
x 2 , and x 3 are selected variables on which x cl , x c2 , and x c3 have been set as 
acceptance criteria. The score profiles of two options, O, and O p may then 
be given as [x n , x a9 x i3 ] and [x J{ , x j2 , x y3 ], respectively. Following the conjunc¬ 
tive model: 

O; Acc, if x n > x cl and x l2 >: x c2 and x l3 > x c3 , (2.1) 

where Acc stands for "is judged acceptable.” When O, and Oj both appear to 
be acceptable, a supplementary decision rule may be: 

Oj. > Oj, if / [x n , x a , x, 3 ] > / [x jU Xj2, jc j3 ], (2.2) 

where .> indicates “is preferred to.” The simplest form of/is, 

Oj. > Oj, if x n > x JX and x a > x j2 and x i3 > x j3 , (2.3) 

but this (strong) dominance rule will not often be applicable in practice. 
More convenient supplemental ways to decide among conjunctively accept¬ 
able options are to sharpen the acceptance criteria used in formula 2.1 or to 
use any of the other satisficing models given below, notably the compensa¬ 
tory weighting model (a primitive version of the weighted-additive utility 
model; see Von Winterfeldt & Fischer 1975). According to the disjunctive 
model: 

Oj Acc, if either x n > x cl or x a > x c2 or x i3 > x c3 . (2.4) 

That is, Oj needs to be “(very) good enough” in only one of the relevant 
variables in order to be judged acceptable. When O, and Oj would both be 
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acceptable following this model, again a supplementary decision rule would 
be required. 

The conjunctive and disjunctive satisficing models may be combined in 
several ways, as, for example when 0, is accepted only if it exceeds x cl and 
either x c2 or x c3y which may both be criteria of ‘'excellence.” Both models in 
fact are extreme forms of the compensatory weighting model, which pre¬ 
scribes the testing of a weighted combination of an option’s scores against 
one overall acceptability criterion (possibly being a weighted combination 
itself): 


O: Acc, if b t x n + + b y x B > C, (2.5) 

where C may either equal + bnX cl 4- b 3 x c3 or be an independent overall 
criterion value. Formula 2.5 implies a simple weighted addition of an op¬ 
tion’s scores on the relevant variables. Various more intricate combination 
rules are possible, but these may not be so easily interpretable from a deci¬ 
sion-theoretic or a psychological point of view. In case 0, and Oj both ap¬ 
pear to be acceptable, formula 2.5 can be transformed into a supplementary 
decision rule: 


0:.> Oj , if biX a + b y x n + b y x B > bpc n + bpcj, 4- b 3 x j3 . (2.6) 

This is the general and currently most popular maximizing decision rule for 
multiattributive, riskless choices, by which all available alternatives may be 
ordered following their relative attractiveness. The assumption is that x u x 2 , 
and jc 3 are (commensurable) utility scales whose values do not interact. 

The three models discussed so far may be used to evaluate single choice 
alternatives that must be accepted or rejected. The following decision rules 
can only be applied to a set of two or more options. 

Elimination-by-aspects (ERA, originally a descriptive probabilistic choice 
model; Tver sky 1972) implies that the relevant variables are first ordered by 
importance and an acceptance criterion is set on each variable. Then all op¬ 
tions are considered with respect to the most Important variable, say x l9 and 
an option G : is rejected when x n < x cl . The remaining options are tested 
next against the second-most-important variable, say jc 2 , and an option 0, is 
rejected when x j2 < x c2 - In this way all (remaining) options are tested against 
all successively less important aspects until only one option is left that has 
(thus far) survived all acceptability tests. Further tests of this option, down 
the remaining order of aspects, w'ould not make sense. 

The elimination-by-aspects model has a “maximizing” cousin in the form 
of the lexicographic rule, in which all variables are ordered by importance 
but no acceptance criteria are set. All options then are judged with respect to 
the most important variable, and the (clearly) most attractive one on that 
variable is selected, whereby the decision process ends. Only if two or more 
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options are (about) equally attractive on the first variable does one proceed 
to evaluate (only) those with respect to the second variable. This process 
goes on as long as two or more options are judged to be equally good on 
successively less important variables. The first option to win out (clearly) as 
one goes along the hierarchy of variables is selected. Thus, the lexicographic 
rule is not a fully maximizing decision rule; it ignores lower-ordered varia¬ 
bles on which once-rejected options may score very attractively. Neither, 
however, is it a genuine satisficing model, as it implies comparative judg¬ 
ment of options on the relevant variables. Thus, it would not be suitable for 
determining whether a given option would be acceptable by itself. Both the 
elimination-by-aspects and the lexicographic decision rules are clearly non¬ 
compensatory. 


Maximizing Models for Riskless and Risky Choices 

This book is about decision analysis and decision-aiding software. Thus, 
the multiattribute utility model and the expected utility principle are getting 
a lot of attention, since these standard models from classical decision theory 
have inspired many software designers. We will therefore keep this section 
short and refer, when necessary, to a few well-known and easily accessible 
textbooks or journal articles. 

To decide among riskless multiattributive alternatives, a straightforward 
model is the compensatory weighting or weighted-additive utility model 
given in formula 2.6, where x ik represents the utility of O, relative to variable 
k. The variables x k (k = 1 . . . r) are customarily taken to be option (or 
rather consequence) characteristics such as income, travel distance, freedom 
of initiative, and variability of tasks associated with a given job option. 
They may also be time periods, distributional targets (e.g., a group of cus¬ 
tomers), or goods obtainable contingent upon outcomes of an uncertain 
event (in which case the outcome probabilities would logically serve as im¬ 
portance weights). In reality, the utility of a decision consequence may easily 
depend on a combination of its inherent characteristics, the time period in 
which it obtains, the person(s) to whom it is allocated, and the contingency 
that produces it. It would not be an easy task to model this formally, let 
alone to assess the various utility components empirically. 

A rather different model for riskless choices is the lexicographic decision 
rule. With this and the basic-weighted-additive utility model, one could 
come up with some interesting and perhaps useful variants. This seems a 
little tricky psychologically, since DMs might lose their understanding for 
what is to be done. The resulting variants may well turn out to be domain 
specific. 

Maximizing expected (multiattribute) utility is the principle of rational in¬ 
dividual choice. To do this, one needs to confront a well-defined decision 
problem having uncertain consequences: anything that may be formally rep- 
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resented as a choice among gambles. This implies that the utilities of conse¬ 
quences can in principle be assessed, as well as the outcome probabilities 
associated with the uncertain event(s). 

Taking the expected utility. 


EU(Oj) = (2 ’ ?) 

/= 1 

(wiiere p i is the outcome probability and u tj the utility of what would be 
obtained if is chosen and outcome i would occur), would subsequently 
make it simple, through rank ordering, to identify the “best” course of 
action. The may, of course, be assessed in a multiattributive way, using, 
for example, the model given in formula 2.6. 

In combination with formula 2.6, formula 2.7 is the foundation of Bayes¬ 
ian decision analysis, a rational methodology, which also pertains to so- 
called predecisional stages (Table 2.1) but which revolves especially around 
option evaluation through utility and probability assessment and with appli¬ 
cation of sensitivity analyses. The methodical instruments of decision analy¬ 
sis are grafted upon several analytical steps (Matheson & Howard 1968; 
Keeney & Raiffa 1976), which may be ordered as follows. 

1. Analysis of goals and objectives, possibly going along with a description of the 
status quo. 

2. Structural modeling of the decision problem. 

3. Probabilistic or uncertainty analysis. 

4. Value or utility analysis. 

5. Optimization, including sensitivity analysis. 

Let us briefly consider each of these. 

In a hierarchical goal (or objectives) analysis , the DM may gradually de¬ 
compose and concretize one or a few general goals into a comprehensive and 
coherent set of specific value or utility attributes. These may later serve to 
evaluate the consequences of given alternatives. But in ill-defined situations, 
they may also suggest the concrete actions that might be undertaken to 
achieve adopted goals. To evaluate suggested options with respect to their 
feasibility, an analysis and description of the current situation may be re¬ 
quired. 

Structural modeling of the decision problem is aimed at clarifying the re¬ 
lationships among current options, uncertain events, possible contingent 
choices, and eventual consequences. Modeling may be done in terms of a 
decision matrix of options by attributes, a decision tree diagram formalizing 
the set of scenarios involved, or an influence diagram (see Schachter 1986; 
Merkhofer 1987, p. 105). 
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During uncertainty analysis , probabilities are estimated and assigned to 
possible outcomes of relevant uncertain events. Either in this stage or condi¬ 
tional on later sensitivity analysis, one may find that the assessed probability 
distributions are not informative enough. The acquisition of additional in¬ 
formation may be considered, the expected value of that information (and 
the choice to collect it) may be determined, and posterior probabilities may 
be computed, with the help of Bayes’s theorem (Edwards, Lindman & Sav¬ 
age 1961; Lindley 1984). During uncertainty analysis the informational basis 
for probability assessment has to be carefully identified, so that it becomes 
clear whether one may rely on the logical, the frequentistic, or the personal- 
istic interpretation of the probability concept (cf. the variants of risky deci¬ 
sions distinguished in Figure 2.2). 

Value or utility analysis serves to assign judgments of relative attractive¬ 
ness to the possible consequences of a set of choice alternatives. This may be 
done in various ways, depending upon how easy (or few-dimensional) one 
considers the task to be and how conscientiously one wishes to evaluate 
(Keeney & Raiffa 1976; Vlek & Wagenaar 1979; Lindley 1984; and von Win- 
terfeldt Sc Edwards 1986 provide systematic treatments of utility assess¬ 
ment). Multiattribute utility analysis may well start with a hierarchical goal 
analysis yielding the specific set of attributes to be used. Or it may be 
founded on a series of triadic comparisons of choice alternatives whereby 
their discriminative characteristics may come to the light (Humphreys & Mc- 
Fadden 1980). Thus, utility analysis may incorporate a specific kind of prob¬ 
lem structuring. Once the relevant attributes have been identified, a simple 
weighted-additive aggregation rule such as formula 2.6 may yield the overall 
utility of a given decision consequence or (riskless) choice alternative. 

Optimization analysis involves the computation of expected (multiattri¬ 
bute) utilities and a rank ordering of alternatives following their relative at¬ 
tractiveness. It also may imply sensitivity analyses during which judgmental 
inputs to the analysis are systematically varied to check the sensitivity of the 
computed preference order for variations in one or more “doubtful” assess¬ 
ments. The need for and the possibility to perform sensitivity analyses pro¬ 
vide powerful reasons for the computerization of decision analysis. In fact, 
a single run through a decision problem is hardly interesting if one knows 
that one can (and should, often) iterate the expected utility calculations with 
repeated corrections and/or adaptations of probability and utility judg¬ 
ments. Sensitivity analysis may also pertain to structural problem elements, 
such as the specific subset of an original list of relevant attributes in a multi¬ 
attribute utility analysis. 

Expected (multiattribute) utility maximization thus functions as the re¬ 
volving door of rational decision analysis. For decision problems involving 
uncertain outcomes, however, there also are some other decision rules. De¬ 
spite their maximizing character, these are usually considered suboptimal, as 
the uncertainties involved are ignored. 
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The maximax payoff rule stipulates that one should first determine the 
maximum possible payoff under each choice alternative and then maximize 
across options while considering only the maximal payoffs. In contrast, the 
maximin payoff rale prescribes first looking at the minimum possible payoff 
per alternative and then maximizing across options with disregard of other 
possible payoffs. The principle of insufficient reason holds that one behaves 
as if all uncertain outcomes are equally likely, so that the “best” alternative 
can be identified by taking the average expected payoff per option. Finally, 
the minimax regret rule first involves a computation of regret values—the 
difference between the highest payoff across alternatives and every other 
payoff, under each uncertain outcome. Next, the maximum regret value per 
alternative is identified, and the “best” option is determined by minimizing 
across options while considering only the maximum regret values. 

As Cooms, Dawes, and Tver sky (1970, chap. 5) have shown, maximax 
and maximin payoff, minimax regret, and “insufficient reason” may lead to 
different “best” choices among a given set of alternatives with numerically 
evaluated consequences. For a truly equiprobable outcome set, the principle 
of Insufficient reason would be equivalent to expected value (or utility) max¬ 
imization under a uniform probability distribution. These rules for decision 
making under Ignorance may well be used as a basis for decision support, 
provided that one is aware of the “evaluative mood” (or meta-strategy) that 
they imply. 


Multimodel Decision Approaches 

As suggested in Table 2.2, “rewarding,” “satisficing,” and “maximizing” 
choice behavior is inherently embedded: most of us frequently face routine 
operational decision problems flowing from regularly occurring tactical de¬ 
cisions; only rarely do we have to take truly novel and complex decisions that 
set the stage for a whole new 7 sequence of tactical and operational choices. 
Thus, a full-fledged decision analysis seems called for only infrequently; 
standardized and automatic decision making are the rule rather than the ex¬ 
ception. 

However, the shifting up and down mode of decision making (Table 2.2) 
occurs not only between different types of problem; it may also occur within 
a given class of decision problems, among different DMs, and even within a 
single decision process. For a given type of problem, such as the selection of 
a job applicant, the reliance on analytic versus synthetic judgment may well 
depend on the importance of potential consequences (a variable not incor¬ 
porated in the taxonomy of Figure 2.2) or on the motivation and seriousness 
of the DM. Important problems invite analytical judgment; motivated and 
serious DMs tend toward the same. But a DM who relies on synthetic judg¬ 
ment may do so through experience, not because he or she is unmotivated or 
lazy. Experience with a particular class of problems drives you down the 
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different modes of decision making: from slow, conscientious analysis to 
fast, effective synthesis. These considerations need to be taken into account 
when selecting a DSM for solving a given decision problem. 

More directly relevant for professional decision supporters is the possibil¬ 
ity of combining a satisficing and a maximizing decision model within a sin¬ 
gle decision process. The satisficing rule would then be used for the 
relatively easy task of screening out acceptable options. This might greatly 
reduce the effort required to complete the process with a subsequent rule, by 
which the remaining alternatives would be evaluated in a comparative, com¬ 
pensatory manner. 

One step beyond this satisficing-then-maximizing decision approach lies a 
modular decision support system that would allow the DM to apply a ra¬ 
tional sequence of decision rules, such as elimination of dominated alterna¬ 
tives, testing options against (strict) acceptance standards, elimination by 
least attractive aspects (such as certain risk dimensions), and expected 
multiattribute utility maximization with respect to the remaining subset of 
alternatives. 

A practical decision philosophy resembling this appears to be developed 
in radiation protection policy (ICRP 1977). Here, the acceptability of a cer¬ 
tain radiological application is determined by applying three “principles”: 
(1) justification: “Are the intended benefits really necessary?” (2) ALARA: 
“Is the radiation level as low as reasonably achievable?” and (3) tolerability: 
“Is the unavoidable radiation level below maximum tolerable limits?” The 
order in which these principles should be applied is still being debated: it 
also seems possible to apply them conjunctively. Radiation protectionists do 
not seem to consider applying them in a compensatory manner, as when a 
higher radiation risk is deemed acceptable so long as the benefits are badly 
needed (which, of course, regularly happens in medical settings). 

The importance of taking a multimodel approach as the basis for design¬ 
ing explicit decision support should not be underestimated. The flexibility 
among different modes and rules of decision making would be appealing to 
many users, if only because it would make the DSM as a whole to reflect 
some basic characteristics of users’ own unaided way of making decisions. 

SOME PROBLEMS AND METHODS OF 
SOCIAL DECISION MAKING 

Rational decision theory, and thus Bayesian decision analysis, is inevita¬ 
bly individualistic. For a given set of alternatives, maximizing expected 
(multiattribute) utility may yield a certain “best choice” for person A, but B 
may come up with another “most preferred” option, and this again may be 
different for person C. Obviously, people have different preferences; this is 
only partially attributable to differences in beliefs and underlying informa¬ 
tion; it is often due to differences in taste; and de gustibus non est disputan - 
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dum (a Latin way of respecting the interpersonal incomparability of 
utilities). 

It is a decades-old finding of social choice theorists (Arrow 1951; Sen 
1970; see also Blair & Poliak 1983; Bezembinder 1989) that there does not 
exist a nondictatorial method for the rational amalgamation of differing in¬ 
dividual preference orders with respect to a given set of choice alternatives. 
This means that any practical social voting method applied to three or more 
options could not be adopted a priori by all parties involved (who would 
declare to “accept” the result whatever this would be), since the result may 
not be paradox free (e.g., it may contain social intransitivies) or it may de¬ 
monstrably be method dependent. 

Three principles of social choice and three practical solutions are sche¬ 
matically pictured in Figure 2.3 (from Vlek & Cvetkovich 1989, p. 308). 
Each solution (lowercase letters) meets only two principles (capital letters) 
while violating the third one (in the opposite corner). Since dictatorship is 
unpopular in democratic communities while consensus seeking is notori¬ 
ously undecisive, majority rule is a widely accepted social decision method. 
This, however, yields a guaranteed social preference order only when two 
options are considered at a time. Thus, majority rule usually amounts to 
voting for or against a proposed course of action, and the order in which 
two or more alternatives-proper are put on the voting agenda may signifi¬ 
cantly affect the selection of the “socially most preferred” option. 

Majority voting, then, usually yields one collectively preferred option, 
which has to be accepted by all those who did not personally rank it as 
“best.” This suggests that prior application of an individual satisficing rule 
by each party may considerably reduce potential social conflict. Individual 


Figure 2.3 

The Trilemma of Social Choice 


COLLECTIVE 

EAXIOHALITY 



rule 

Note: Represented by three principles (solid triangle and capital letters) and three practical so¬ 
lutions (dashed triangle and lowercase letters), each meeting only a pair of them. 
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satisficing would reduce the set of alternatives to be subjected to majority 
voting, provided that all parties agree to exclude from voting any alternative 
that is (strictly) unacceptable to any party. 

Given the individual decision support schemes of Figure 2.1 and Table 
2.1, what could one undertake to support social decisions in groups and or¬ 
ganizations? First, most social decision problems are sufficiently ill defined 
for social choice theory (which pertains to given choice sets) to be only par¬ 
tially relevant. Our multicomponent scheme of decision making would seem 
to apply a fortiori to group and organizational decision making, where pro¬ 
cesses of information acquisition and exchange and of problem structuring 
very much condition the set of choice alternatives to be socially evaluated. 
Thus, again the question arises, Which component of the total process 
would need to be supported most? 

Second, since there is no collectively rational decision method, one would 
think that a pragmatic approach could be useful, in which the judgments of 
substantive experts and the preferences of those potentially affected by a de¬ 
cision’s consequences are brought together under the coordination of some 
ultimately responsible decision maker (maybe a professional “knot cutter”). 
On a small scale this could, for example, yield concrete solutions to difficult 
doctor-patient (or doctors-patient-family) decision problems in medical set¬ 
tings. On a larger scale, as in connection with new technological systems or 
processes, one might have to go by a well-organized social decision proce¬ 
dure in which experts, special interest groups, and public representatives 
work together in a series of meetings aimed at exchanging information, 
structuring the decision problem, and (somehow) collectively evaluating the 
available choice alternatives. (Vlek & Cvetkovich 1989 also summarize and 
comment upon several recent approaches proposed by others. Friend & 
Hiekling 1987 provide an artistic exposition of their strategic choice ap¬ 
proach.) 

Third, we should be aware of the fact that social decision makers must 
overcome two general problems: (1) each party has an individual judgment 
problem, and (2) all parties together face a social interaction and aggrega¬ 
tion problem. Thus, effective social decision support may partly consist of 
methods and rules taken from individual decision theory; partly it should 
supplement these with (or incorporate them into) a set of procedures and 
techniques for information exchange and communication among parties, 
for aggregate problem structuring, and for joint evaluation of feasible 
courses of action. 

Naturally, the individual decision quality criteria should be supplemented 
with—or partly replaced by—criteria for “good” social decisions. Here one 
would especially think of consensus measures regarding what would be rele¬ 
vant information, an adequate problem structure, and valuable conse¬ 
quences. Also important would seem the extent to which the supported 
decision procedure is capable of building trust in the course of action even- 
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tually selected. A trusted social decision procedure may strongly encourage 
individual parties to swallow their initial objections to a chosen alternative 
they do not or did not like. 

We cannot ignore the voluminous literature on organizational decision 
processes (see, e.g., Bass 1983; Wright 1985, pt. 3). What lessons for 
decision supporters can be learned here? Descriptive research has yielded 
the recurrent conclusion that complex decisions in organizations tend to 
be taken along several recognizable stages. For example, Mintzberg, 
Raisinghani, and Theoret (1976) described twenty-six important decision 
processes as following problem identification, development of alternatives, 
and evaluation and selection. In a more elaborate descriptive study, Nutt 
(1984) analyzed seventy-three organizational decision processes following a 
five-stage model: (1) problem formulation, (2) concept development (of al¬ 
ternatives), (3) detailing (of alternatives’ workability), (4) option evaluation, 
and (5) implementation of chosen plan. Such descriptive stage models line 
up well with the prescriptive model of process components sketched in 
Table 2.1. 

A special feature of Nutt’s (1984) study is the varying operation of three 
distinct steps within each of the several stages —search, synthesis, and analy¬ 
sis. Using his five stages and three steps-within-stages as a coding scheme, 
Nutt was able to characterize each of the seventy-three decision processes as 
belonging to one of five archetypal processes: 

1. Historical model: Do what has been demonstrated to “work.” 

2. Off-the-shelf: Choose among the best available plans. 

3. Appraisal: Conduct some kind of decision analysis. 

4. Search: Look around and wait for a full-blown good idea. 

5. Nova: Mobilize everything to create an innovative plan. 

These categories run fairly parallel to the levels of decision making distin¬ 
guished in Table 2.3. Apparently, decision makers in organizations do take 
different approaches—which imply different quality guarantors (see Nutt 
1984)—to decision problems of differing nature. A conclusion like this may 
pave the way tow ard fruitful utilization of descriptive findings for the design 
of effective organizational DSMs. 

In modem life, of (purely) individual and social decisions the latter seem 
to be the rule rather than the exception. For professional decision sup¬ 
porters, this observation may be strengthened by the thought that supporta¬ 
ble decision problems—the strategic ones calling for explicit analysis (Table 
2.2) —are often tackled by several decision makers simultaneously (at least 
by you and your adviser). Therefore, the development and empirical evalua¬ 
tion of "good” social decision methods seems to be a crucial task for deci¬ 
sion theorists who want to aid society in solving its numerous and diverse 
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decision problems. “No coherent approach exists,” says Lindley (1985, 
p. 180). “Yet the need for one is more urgent now than at any point in the 
history of the world.” We agree. 

CONCLUSIONS, SUGGESTIONS, AND SOME LOOSE ENDS 

In this chapter, we have attempted to explicate the idea of decision sup¬ 
port, mostly from a decision-theoretic and a cognitive-psychological point 
of view. Although we have tried to develop our ideas step by step and have 
covered a lot of ground, we still feel somewhat uneasy. This may have to do 
with the notion of decision making itself. It seems to be so variable and so 
task dependent that it is hard to define it usefully in other than a fairly re¬ 
stricted sense (e.g., as applying to choices among gambles). The definition 
of a decision problem seems general enough that almost any approach to¬ 
ward solving a decision problem could be called a decision support method. 
Actually, this is a major difficulty in the growing field of (management) de¬ 
cision support systems defined as: “computer-based systems, that help deci¬ 
sion makers, confront ill-structured problems, through direct interaction, 
with data and analysis models” (Sprague & Watson 1989, p. 2; see also 
Mclean & Sol 1986). 

Our own view is that decision making follows a varying mode of cognitive 
processing, depending on the strategic, tactical or operational nature of the 
problem (cf. Table 2.2), and that complex decision making consists of infor¬ 
mation acquisition, problem structuring, and option evaluation (cf. Figure 
2.1). Thus, adequate decision support should be specific to the level or mode 
of decision making, and it may emphasize any one or all three components 
of Figure 2.1. 

There are debates about the value of expert systems vis-a-vis procedural 
aids for complex decisions (e.g., Fox 1984; Keeney et al. 1988). It would 
seem that the information acquisition (and storage and retrieval) component 
of decision support is being captured by expert systems, whereas the prob¬ 
lem-structuring and option-evaluation components are incorporated in pro¬ 
cedural decision aids. Also, however, expert system designers working with 
if-then rules are bound to focus on frequently occurring, relatively low-level 
operational or at most tactical decision problems, on which sufficient expe¬ 
rience has been and is being accumulated. In contrast, procedural decision 
supporters primarily work on complex, strategic decisions where problem 
structuring and option evaluation are done under a great deal of missing 
information. 

It is a challenge for future research and DS design to integrate substantive 
(knowledge-based) and procedural decision support such that the DM 
would not have to operate in a frozen task environment and follow a rigid 
mode of decision making. The ideas of Miller, Galanter, and Pribram (1960) 
and of Toda (1976, 1981) are highly relevant here. A recent computer pro- 
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gram that embodies this kind of integration for multiattribute utility analy¬ 
sis is DECMAK by Bohanec and Rajkovie (1988; see also Rajkovic, Bohanec 
& Batagelj 1988; Bohanec and Rajkovic, this book). 

Designing methods to help people make better decisions than they could 
do by themselves constitutes the litmus test of decision theory. It appears 
that this test cannot be administered without the availability of models and 
modes of “good” decision making, sufficient knowledge about the nature of 
unaided decision processes, of individuals and in groups and organizations, 
and a set of practical criteria for evaluating the quality of decisions. In the 
past, decision theorists and researchers have made gradual progress toward 
fulfilling these conditions. The task now is to bring normative and descrip¬ 
tive ideas and findings together and to set out on a prescriptive course firmly 
supported by empirical studies. The application domains are there: in com¬ 
pany management, in the educational system, in technology assessment and 
environmental protection. In medical practice and health care, and perhaps 
in the judicial system. 
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A Psychological Process 
Perspective on Decision Making 


Ola Svenson 


This chapter theorizes about cognitive processes preceding decisions made 
by single individuals. In doing so, a psychological process tracing perspec¬ 
tive (cf. Svenson 1979; Montgomery & Svenson 1989a; Ford et al. 1989; 
Svenson 1992) will be adopted for presenting evidence and theories for un¬ 
aided decision making. Readers who want to supplement this perspective 
with research initiated from other perspectives are referred to other sources, 
such as Kahneman, Slovic, and Tversky (1982), Abelson and Levi (1985), 
Arkes and Hammond (1986), von Winterfeldt and Edwards (1986), Rohr- 
mann et al. (1988), Dawes (1988), and Bell, Raiffa, and Tversky (1989a). 

The process of making a decision starts with the appearance of a decision 
problem and the realization of the need for a decision. It continues with the 
identification and/or construction of decision alternatives, followed by 
the decision process itself. Because the initial stages are very important for 
the following decision process, they will also be examined in this chapter. 
Therefore, in a broader sense, a decision may be defined as a psychological 
process starting with the realization of the need for a decision, going 
through the implementation of the decision process, and ending with the 
registration of the outcome of the decision. The outcome part of a decision 
is important because it constitutes part of the input for later decisions. 

For those active in designing decision support systems, it is important to 
learn about nonsupported decision processes. This is because any decision 
support must be compatible with the unaided decision process if any 
progress is expected. First, compatibility must be achieved for reasons of 
communication during the decision-aiding process. Second, most decisions 
reached through decision-aiding techniques are evaluated partly in relation 
to unaided opinion about the proposed solution of the decision problem. 
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TWO TYPES OF DECISIONS 

Most of our lives seem to follow logically from past decisions and present 
circumstances. A decision problem that appears along this path represents 
one of two possible generic types of problems. In the first, type A, the deci¬ 
sion maker arrives at a point where the future branches into two or more 
paths, each forming a decision alternative. In this situation, it is not possible 
to remain at the status quo or even follow the path that seems more natural. 
The decision maker’s task is to choose one of these options, all of which are 
new alternatives not following more or less automatically from the past. To 
give a simple illustration, one has to decide on what to eat when in a new 
restaurant, and often this decision does not follow automatically from the 
past without any deliberation. 

In type B decisions, the decision maker may experience his or her present 
and projected future situation as unsatisfactory and starts working for a 
change. In this case, doing nothing to change the situation is one decision 
alternative (status quo), and possible changes constitute other decision alter¬ 
natives. A type B decision may also be exemplified by a decision in which 
the present situation is acceptable but an attractive new alternative emerges 
(e.g., an attractive job offer). A type B decision often precedes a type A de¬ 
cision. For example, the decision maker may first decide to move to a new 
place (type B decision), where he or she must choose a new job from a set of 
alternatives not including his or her earlier profession (type A decision). 

In the decision research literature, most decisions are modeled as type A 
choices among existing choice alternatives (cf. Abelson & Levi 1985). Of 
course, theoretically; one of the alternatives could be regarded a status quo 
alternative in the type B case. However, from a psychological point of view, 
the difference between the two types of decisions should not be ignored, as it 
may simplify decision theory. First, in a type B situation, the “no action” 
alternative may function as a reference alternative, which the other alterna¬ 
tives are compared against. Second, the choice of this alternative is often 
seen as needing fewer psychological resources than other alternatives (fewer 
transaction costs). Adherence to the already implemented course of action is 
psychologically a low-cost alternative, which may lead to decision inertia. In 
other words, one may stick to the old decisions and policy, hoping that it 
will work out, although it at some level may be perceived as suboptimal (cf. 
Staw* 1976; Staw r & Fox 1977; Laughhunn & Payne 1984; Arkes & Blumer 
1985; Samuelson & Zeckhauser 1988). Another and related explanation to 
decision inertia is a reluctance to admit that the decision maker made a sub- 
optimal decision in the past. On the other hand, human life may be seen as a 
balance between conservatism and a need for change. Therefore, forces 
counteracting decision inertia may also be activated in many situations. If 
type A and type B decisions are differentiated, it would be no surprise that 
they 7 do not activate the same decision processes. 
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From a psychological perspective, decisions between, for example, gener¬ 
ally good alternatives may differ from choices among bad alternatives 
(Payne, Bettman & Johnson 1988; Tversky & Kahneman 1981; Maule 1989) 
with different decision rules for different types of decisions. Usually norma¬ 
tive decision theory does not allow for this, but decision-aiding techniques 
somehow have to accommodate for the fact that in some situations decision 
makers do not want to apply normative decision theory, even if they are in¬ 
formed about how to make rational decisions (Adelbratt Sc Montgomery 
1980). 

Few researchers have studied the first stages of a decision process in which 
the decision maker becomes aware of the need of a decision. This is interest¬ 
ing because the first stages may be decisive for the following choice. Also 
interesting are the many cases in which people are not aware of when what in 
retrospect appears as a really important decision was actually made. At the 
time, the decision was seen as minor, but it may have developed into a major 
decision through many automatic subdecisions. Detecting or not detecting 
an important decision problem when it first appears seems to be one of the 
most fundamental aspects of high-quality decision making. This was illus¬ 
trated on the societal level by Ingelstam (1980), who analyzed decisions 
closely knit to industrial complexes (e.g., the traffic road automobile indus¬ 
trial complex and the energy-industrial complex). According to Ingelstam, 
“important societal decisions no longer present themselves in a clear-cut 
way. . . . They do not present themselves at all” (p. 266). Janis and Mann 
(1977) presented a model in which the first stage in the decision process was 
the realization of the need of change. In an empirical study, Karlsson (1987) 
found that a majority of the important real-life decisions reported by his 
subjects were of type B and standard situations (type A), with several alter¬ 
natives in minority. 

CREATING DECISION ALTERNATIVES 

Once a decision problem has been heeded, this does not mean that all 
available options follow automatically. In type A situations, it is clear that 
several alternatives exist, but in type B situations no decision alternative ex¬ 
cept the status quo in the form of an extrapolation from the present can be 
assumed. 

Even if there are several alternatives, those existing are sometimes ill de¬ 
fined, and additional alternatives may have to be created. The initial formu¬ 
lation of a decision problem is assumed to be very important for the final 
decision. First, the construction and early scanning of decision alternatives 
are significantly related to the final decision (Montgomery & Svenson 
1989b). In deciding what move to make in chess, good players differ from 
weak ones in terms of generation of alternatives because they have already 
included the finally chosen move in the first set of alternatives (de Groot 


72 


Computer-Aided Decision Analysis 


1965). Second, the way in which formally equal alternatives are described 
may be decisive for the final choice. The striking effect of “framing” illus¬ 
trates this very convincingly (cf. Tversky & Kahneman 1981). Here, Mont¬ 
gomery and Svenson (1989a, p. 130), distinguish between objective and 
subjective framing, of which the latter experienced context is effective in 
producing widely diverging decisions across subjects and across formally 
equal decision problems. Decision alternatives appear in space and time. For 
the decision maker, alternatives may phenomenologically seem like scenarios 
or casual chains of events into the future (Axelrod 1976; Abelson 1981; 
Schoemaker 1988). Even quite abstract numerical laboratory tasks may be 
elaborated by the subjects, who may add hypothetical future developments 
to quite abstract options. This may be done to make the task more concrete 
and real (Svenson 1985). 

All the possible contingencies of a decision tree are not explored by the 
decision maker. Because of cognitive limitations, individuals are more likely 
to explore a few of the possible paths through a decision tree while con¬ 
structing and exploring decision alternatives. Decision theory projects the 
future onto the present, sometimes with explicit discounting of value 
(Fishbum 1970), when describing alternatives. People must do the same 
sometimes, but the time dimension may be simply neglected or be explicit 
for some alternatives only. 

In this chapter, decision alternatives are represented in the following way 
for both type A and type B decisions. An alternative (e.g., a trip) is charac¬ 
terized by different attributes (e.g., cost, temperature at the destination). Al¬ 
ternatives are characterized by physical or objective characteristics (e.g., cost 
in SEK, temperature, size of hotel room) having a subjective correspondence 
in perceived aspects (e.g., perceived size of room). When making a decision, 
those aspects are given subjective values on scales of attractiveness, one for 
each of the attributes. This representation of the alternatives constitutes the 
psychological decision problem space. A value system is a common concept 
for those principles used for ordering the aspects on an attribute in a prefer¬ 
ence order. Montgomery (1989) provides a recent discussion of values in de¬ 
cision making, as do Bell, Raiffa, and Tversky (1989). 

My definition of a decision problem is in general accordance with decision 
theory’s dimensional representation (Keeney & Raiffa 1976; von Winterfeldt 
& Edwards 1986). However, in contrast to most other decision theoretic rep¬ 
resentations, it does not assume commensurability of attractiveness across 
attributes or more than an a priori rank order representation of attractive¬ 
ness of the aspects within an attribute (Svenson 1979). Like most other deci¬ 
sion theoretic representations, this approach has difficulties in representing 
decision alternatives perceived in other and more holistic ways. For example, 
a piece of art or another individual may give holistic impressions that are 
not well suited for dimensional atomistic or feature analysis. Still, the repre- 
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sentation seems well suited for forming the link between decision theory and 
more purely descriptive psychological representations of decision problems. 

THREE DESCRIPTIVE MODELS OF DECISION MAKING 

Descriptive process models of decision making include models in which 
the decision maker applies a set of decision rules to the alternatives to 
choose from (Montgomery & Svenson 1976; Payne 1976, 1982; Svenson 
1979). More recent process models (Montgomery 1983; Montgomery & 
Svenson 1989a) also include changes in the evaluative representation of the 
decision alternatives during the entire decision process. This continuously 
ongoing restructuring of a decision problem may be driven by goals such as 
a wish to reach a decision with a minimum of conflict or to reach a decision 
that can be justified for one’s self and others. One particular structure of a 
decision problem, that of dominance, meets several psychological require¬ 
ments of an acceptable decision. In several investigations, it has been shown 
that cognitive processes in decision making are compatible with a search for 
dominance model (Montgomery & Svenson 1989a). This has been illustrated 
through subjects’ information search and restructuring of alternatives dur¬ 
ing the decision process so that the chosen alternative comes closer to domi¬ 
nating its most serious competitor toward the end of a decision process than 
it was at the beginning (Montgomery & Svenson 1989b; Dahlstrand & Mont¬ 
gomery 1989). Other recent formulations (Beach & Mitchell 1967) have fo¬ 
cused on representations of decision alternatives, which are similar to scripts 
or scenarios. 


The Dominance Search Model 

Decisions are easy to make when one alternative dominates the other(s) — 
that is, when it is equal or better on all attributes and better on at least one 
attribute than its competitor(s). Montgomery (1983) focused on this in his 
dominance search model of decision making. The most important aspect of 
this model is that it describes the decision maker as changing his or her rep¬ 
resentation of a problem during the process of making a decision. More spe¬ 
cifically, the decision maker’s evaluation of the chosen alternative is 
modified in such a way that it appears to be closer to dominating its most 
serious competitor toward the end of the decision process. 

The model contains four main phases. In the first, the preediting phase , 
the decision maker selects alternatives and attributes to consider in the deci¬ 
sion process. This phase is related but not identical to the editing described 
by Kahneman and Tversky (1979). This phase is also related to the creation 
of the decision alternatives stage described earlier. However, the creation of 
alternatives stage does not presuppose an attribute jc alternative representa- 
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tion. And it is not as closely tied to the final representation of the decision 
alternatives as in the preediting phase of the dominance search model. 

The second phase according to Montgomery concerns finding a promis¬ 
ing alternative . Here, one alternative is picked out that seems to have a 
chance of becoming dominant over the others. The reasons for this may be 
either evident or unclear to the decision maker. 

After this, the dominance testing phase follows. The promising alterna¬ 
tive is tested against the others to see if it dominates the other alternatives. If 
this is not the case, the fourth phase, the dominance structuring phase , 
stans. Now the decision maker tries to change the decision problem in vari¬ 
ous ways to achieve dominance. Put in decision support terminology, the 
decision maker performs a sensitivity analysis on his or her attractiveness 
representation, but the analysis is made with a specific purpose: to find out 
whether a decision under dominance can be structured so that the promising 
alternative stays within the acceptable uncertainty bounds of the decision 
maker’s mental representation of the situation. 

Image Theory 

Like most process tracing decision research, image theory was presented 
in response to doubts about the descriptive competence of behavioral deci¬ 
sion theory. Beach and Mitchell (1988) recently published the main theoreti¬ 
cal framework for this descriptive model of decision making. 

In image theory, traditional multiattribute representations are substituted 
for broader concepts. Images are cognitive representations or schemata 
claimed to be specific to decision behavior. An image represents the decision 
maker’s ideals or principles in some field of decision making. It also in¬ 
cludes goals and what to do to reach those goals in that particular field of 
decision making. To specify, decisions are seen as the result of the utilization 
of four different images. 

First, the self-image represents the decision maker’s basic values, morals, 
and behavioral codes —principles regarded as desirable without much fur¬ 
ther thought. Second, the trajectory image represents the decision maker’s 
Immediate and remote goals temporally ordered and forming a plan for the 
future. New goals may be included in this image through evaluation against 
the self-image and the current trajectory image. Third, the action image rep¬ 
resents the different action plans implemented for reaching the goals on the 
trajectory image. Fourth, the projected image consists of events and states 
that the decision maker predicts what will follow if the current action plan is 
carried out. 

It is clear that this represents a different modeling of decision-making 
processes than those usually employed. In principle, however, it seems possi¬ 
ble to model much of the above in a carefully chosen multiattribute model. 
But in contrast with multiattribute decision theory, a decision process also 
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includes the adoption or rejection of goals, as well as contingent evaluation 
of current progress (which perhaps could be related to control theory). The 
reevaluation of current goals is similar to some of the thoughts of other 
models of decision making (cf. Svenson 1992; Montgomery 1983; Mont¬ 
gomery & Svenson 1989a). 

Although image theory has not been explicitly presented as a process 
model, itjias characteristics that place it in that category (e.g., the continu¬ 
ous monitoring of the projected image so that it does not deviate too much 
from the trajectory image). The model seems not yet elaborate enough to 
provide a full-blown alternative in itself. Still, it contains many interesting 
ideas that might both complicate and enrich the design of decision-aiding 
software. 


Differentiation and Consolidation Theory 

Recently Svenson (1992) presented the differentiation and consolidation 
theory of decision-making, which models human decision making as an ac¬ 
tive process in which one alternative is gradually differentiated from other 
available alternatives. The theory is based on the assumption that it is not 
sufficient to choose the best alternative, but that a decision involves the se¬ 
lection and creation of a candidate that is sufficiently superior for a deci¬ 
sion. This is achieved in differentiation processes that are holistic, structural 
or process related. To exemplify, structural differentiation includes both 
facts and attractiveness restructuring. 

Following a decision, the theory predicts consolidation processes that 
work in favor of the chosen alternative. Both differentiation and consolida¬ 
tion are driven by the fact that through experience with the unpredictability 
of the future, a decision maker has learned to prepare for threats against the 
chosen alternative. The further this alternative has been differentiated and 
consolidated, the less the risk of post-decision ambiguity, regret or decision 
reversal. 

Empirical evidence for the theory has been published by Svenson and 
Malmsten (1991) and Svenson and Benthorn (1992). Differentiation and 
consolidation theory includes many other decision theories that may be re¬ 
garded as special cases if they are further developed to fit the framework 
provided by the new theory. 

LEVELS OF EXPLORATION OF VALUES AND 
DECISION PROCESSES 

Decisions are based on the decision maker’s values, which are mapped 
onto the decision problem. The extent of the depth of the search and 
processing of values differ as a result of, for example, earlier experience that 
the decision maker may have of a particular situation and its perceived im- 
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portance. The following distinctions are related to how a decision situation 
is apprehended and to how much in depth a decision maker explores his or 
her own value system in order to reach a decision. At the lowest level (1), no 
explicit mapping of the value system onto the decision problem is per¬ 
formed, while at the highest level (4), elaborate mapping of values and a 
creative process involving structuring and restructuring of the problem pre¬ 
cede the decision. The borders between the different levels of processing for 
a decision are not as sharp as the identification of the four levels seems to 
indicate. Rather, the levels indicate different approximate intervals on a con¬ 
tinuum from no to a high degree of information processing for mapping the 
decision maker’s values onto the decision problem he or she is about to 
solve. 


Level 1 Decisions 

At the first level, the decision maker recognizes the situation as similar to 
an earlier one. He or she does not refer to any values at all, although the 
decision strategy may be traced back to a value system that determined the 
decision the first time it was made. This is because it is possible to use, for 
example, earlier successful decisions as reference alternatives, which then 
represent a positive value. In this way, no explicit reference to the decision 
maker’s value system is needed. 

Decisions on the first level include processes other than comparisons of 
values. Processes such as similarity matches between current alternatives 
and alternatives chosen in the past dominate the psychological information 
processes. The alternative most similar to an earlier successful decision al¬ 
ternative is chosen without any reference to values in the comparison 
process. Alternatives that are similar to earlier nonchosen alternatives or un¬ 
successful decisions may also be eliminated without any reference to any val¬ 
ues at the moment. Another type of decision on this level is represented by 
decisions conforming to a trusted other. That is, the decision maker makes 
the same choice as somebody whom he or she trusts and who holds the same 
values. Such decisions may be made following the trusted other’s real deci¬ 
sions, or the decision maker may imagine what the trusted other would de¬ 
cide if she or he was in the same situation. 

Level 1 decisions lead to a final choice in many situations, but they may 
also be seen as a first stage in a more complex decision process. In the latter 
case, this type of decision may produce promising or reference alternatives 
used in later stages of, for example, a dominance structuring process. Such 
alternatives are Important also in more elaborate decision processes. 

Exemplifying a decision on level 1, the decision of what car to buy may be 
determined by an earlier decision. The reasoning and justification may be as 
follows: “When my old car finally broke down completely, 1 decided to buy 
a new Volvo because it is most similar to the old Volvo I had for years.” 
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Level 2 Decisions 

At this level the decision maker feels that he or she makes a decision with 
reference to preferences on one or a few attributes. While at the first level no 
reference is made to evaluation, the second level also involves a routine eval¬ 
uation of the aspects with reference to how the attributes are generally val¬ 
ued. To exemplify, the evaluation that a low price is good is used, as well as 
the higher attractiveness of a bigger size package in the supermarket. In the 
case of the decision to buy a car, it could be an Oldsmobile (because it is the 
biggest available) in one situation and the closest available Japanese car (be¬ 
cause it is most reliable) in another situation with another decision maker. 
Note that decision processes at a given level also include lower-level pro¬ 
cesses. 

Decisions at the second level represent stereotypical and static mappings 
on existing valuations of the manifest aspects on attributes characterizing 
the alternatives (e.g., “biggest is best”). This is done without any further 
explorations into why those values are held. At this level, no new trade-offs 
between attributes may be performed. No explicit conflicts are present. De¬ 
cisions are based on one or perhaps a few salient attributes jointly favoring 
one alternative. Noncompensatory decision rules determine level 2 deci¬ 
sions. Only earlier formed stereotyped trade-offs may be executed in reach¬ 
ing a decision. In order to get along in life, it seems necessary that most 
frequent decisions involve only level 1 and level 2 processing. 

Another type of level 2 decisions are those in which some very quick emo¬ 
tional or motivational process determines a decision (or the promising alter¬ 
native for a higher level decision). Zajonc (1980) has discussed such 
decisions based on feelings rather than cognitive inferences. 


Level 3 Decisions 

At this level, there exists a nonautomatic correspondence between the de¬ 
cision maker’s value system and the aspects describing the decision alterna¬ 
tives. Instead of just stating to themselves, “The more of this attribute, the 
better,” decision makers try to relate attributes to their own value systems. 
Decisions at this level may also use trade-offs between the attractiveness of 
aspects on different attributes or transformations of attributes into new ones 
(e.g., cost per square meter replaces the two constituents). Note, however, 
that no new decision alternatives are created in processing at level 3. To ex¬ 
emplify, a decision at this level may be motivated through “I decide to buy 
this car over that car because I find the difference in price is not so impor¬ 
tant and I value its superior driving characteristics so much that they out¬ 
weigh its poorer exterior looks.” All of the level 3 processing presupposes 
access to values that may be regarded as meta-values to the value system 
used for levels 1 and 2 decisions. Active dominance structuring may be 
found at this level of decision making and higher. 
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Most of this chapter (and this book) concerns decisions made at level 3 or 
higher. When decisions are made at the first two levels, the decision maker 
sees no apparent need for decision-aiding techniques. Note, however, how 
important it may be, for example, in business or in politics, to realize when 
it is not sufficient to work at levels 1 and 2 because simple decisions may 
lead to complex consequences. 


Level 4 Decisions 

At the fourth level, the decision maker faces a new and unfamiliar deci¬ 
sion problem in which the alternatives have to be elicited or created. The 
decision maker now goes beyond the immediate present in terms of values 
and facts. Concerning facts, the decision maker may extrapolate and elabo¬ 
rate the alternatives presented and/or create totally new ones. Here, decision 
theory connects to the psychology of problem solving because parts of the 
constructed decision alternatives may be derived in problem-solving pro¬ 
cesses. For example, different possible ways of attaining a specific goal (to 
get a better car, say) may involve solving the problem of how to make the old 
car perform better (one alternative) and selecting criteria for the selection of 
an alternative used car (to buy). Future scenarios like those presented, for 
example, in image theory and other elaborations of the decision problem 
may also occur at this level. Concerning values, the decision maker actively 
relates the present to his or her value system when creating new alternatives, 
testing their realism, and determining their attractiveness. 

Most people use all four levels of decision making. Some may be engaged 
in level 4 decision making less often than others. Montgomery (in press) dis¬ 
tinguished between simplification and elaboration in decision making that 
may be related to level 2 versus levels 3 and 4 decisions. Again, it is impor¬ 
tant to note that, as a rule, lower-level processes are also included as subpro¬ 
cesses in higher-level processing. 

The more frequent, w r ell learned, and routinized a decision is, the further 
down the hierarchy it is likely to be pushed. This is because decisions lower 
in the hierarchy are associated with less cognitive effort. Hence, energetic 
resources may be reallocated from former, more demanding decisions. 

Type A decisions, in which a choice between a set of decision alternatives 
emerging in front of the decision maker, may involve those from all levels, 
including level 4 (in which new alternatives and new mappings of basic val¬ 
ues on the current problem may have to be performed). What the decision 
maker perceives as more important decisions may be assumed to require 
higher-level processing in both type A and type B situations. If an impor¬ 
tant decision is made at, say, level 2, it may be harder to face and justify the 
consequences if they turn out unfavorably. 

Because type B decisions involve considerations or decisions to leave the 
natural extrapolation of the past, such decisions may or may not involve 
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higher-level processing. However, at least level 2 processing is needed in 
type B decisions, and as the decisions become more important and involv¬ 
ing, level 3 or 4 processing is likely to occur. 

Lack of resources, for example, induced by time pressure, is assumed to 
favor level 1 and level 2 decision making. This is corroborated by findings 
that time pressure leads to greater popularity of noncompensatory decision 
rules than compensatory rules (cf. Christensen-Szalanski 1980; Billings & 
Marcus 1983). However, under even at least moderate time pressure, more 
cognitive resources may be mobilized if the decisions are important, and the 
processing may therefore be performed at the same or even at a higher level 
than usually (Wallsten & Barton 1982). 

The levels of processing may be compared to the cognitive control issue 
and the repeated versus unique decision distinction. To exemplify, Rasmus¬ 
sen’s skill-, rule-, and knowledge-based classification of control behavior 
(cf. Rasmussen, Duncan & Le Plat 1987) may be compared to the present 
levels if it is assumed that consulting one’s value system requires mental ef¬ 
fort and that therefore less effort-demanding routines are developed for 
pushing the processing of repeated similar decisions down to lower levels. 

A PROCESS MODEL OF INDIVIDUAL DECISION MAKING 

The process model presented here contains all levels of decision making. 
Specifically, it elaborates level 4 and 3 decision processes because such deci¬ 
sions are considered more interesting in a decision-aiding context and be¬ 
cause they are fundamental to lower-level routinized decisions. For example, 
many level 1 and 2 decisions are replications of the decision maker’s own or 
others’ decisions which in the beginning were treated as decisions at more 
complex levels. 

Figure 3.1 presents a flowchart illustrating the main stages of a decision 
process. The process starts at the top and ends with a decision. To the far 
right in the figure, the vertically extended box represents the decision 
maker’s value system for mappings of the current problem and a monitoring 
system storing the meta-level strategies governing the decision process. Con¬ 
sultations of this function may take place at almost any stage of the decision 
process, as indicated by the dashed lines. The value system contains both 
more intellectual values (e.g., norms and cultural values) and more funda¬ 
mental physiologically, emotionally, and socially related values (associated 
with e.g., food, sex, or security). The value system may be spontaneously 
and automatically activated (Zajonc 1980) or more deliberately consulted. 

Following the start of the process, the decision maker may face a type A 
(several options emerging at the same time) or a type B decision (realizing 
the need of change and creating or observing alternatives branching away 
from the doing nothing alternative). In both cases, planning of how to con¬ 
duct the decision process will follow. 


Figure 3.1 

A Process Model of Decision Making 
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Planning may be explicit or implicit and unconscious and includes deter¬ 
mining the level at which a decision should be made, the goals to be attained 
in the situation, and the strategy for reaching those goals. All of these com¬ 
ponents of planning are interrelated and determine in part what decision 
rules and restructuring will be used in the processing of the information 
about the decision alternatives. 

The level at which a decision problem is addressed may be determined by 
such factors as experience with the situation, judged importance, difficulty 
of situation, and resources available. However, the classification of a deci¬ 
sion problem should not be regarded as invariably fixed. Instead, it may be 
continuously changing because decisions are made in a changing world and 
because the process of solving a decision problem involves continuous re¬ 
structuring and reappraisal, both before and after the decision has been 
made (Montgomery & Svenson 1989a; Festinger 1964; Lawler et al. 1975). 
To a great extent, decision making is a process contingent on the particular 
information just processed (Payne 1982) but within the general frames pro¬ 
vided by the value system and the goals evoked before a particular problem 
was presented. 

Search for information follows planning. As the arrow going back to 
pl annin g indicates, the way in which search is performed also depends on 
how the information is displayed. When, for example, the information is 
organized either by attribute (e.g., price for all products first) or alternative 
(e.g., one purchase offer at a time), subjects sometimes change their 
processing accordingly (Bettman & Kakkar 1977; Herstein 1981) and some¬ 
times they do not (Sundstroem 1989). One important difference between the 
former and latter studies was that the information was available all the time 
and could be reinvestigated at any time in the former studies but only once, 
when first acquired, in the latter. In conclusion, when the information has to 
be memorized, the presentation format seems to affect the information 
search pattern less than when the information is available all the time. 

The creation of alternatives is an important aspect of decision-aiding 
techniques but takes place for level 4 unaided decisions only. Actually, most 
decision-aiding techniques should regard a decision problem as a situation 
in which new alternatives are first created. Even if this is done just for imme¬ 
diate rejection, it gives a firmer anchoring in both reality and in the person’s 
value system, which is advantageous for reaching the final decision. A refer¬ 
ence alternative may sometimes be identical with the promising alternative 
already presented in conjunction with the dominance search model. The ref¬ 
erence alternative functions as an evaluative anchor in the mind of the deci¬ 
sion maker, and it may be abstract or concrete. An abstract reference 
alternative consists of a set of limiting aspects on a set of attributes. A set of 
aspiration levels (Tietz 1983; Lopes 1987) illustrates this because the decision 
maker knows what he or she wants on the different attributes but has no 
concrete gestalt of such an alternative. While an abstract reference alterna¬ 
tive is based on evaluations of attributes, a concrete reference alternative is 
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based on an existing or imagined alternative forming a gestalt in itself. Re¬ 
cently, Bell, Raiffa, and Tversky (1989b) pointed out the importance of ref¬ 
erence alternatives and some consequences for decision-aiding techniques 
not only for level 4 decisions. Here, more or less complete scenarios or men¬ 
tal representations like those handled by image theory may be applicable as 
descriptions of the reference alternative in addition to aspect times attribute 
representations. When comparisons are made for eliminations or in tests of 
the promising alternative (later in the flowchart), the reference alternative 
may be consulted, the levels on the specific attributes read off and compared 
to another alternative. Montgomery (1989) discusses anchoring in decision 
processes and refers to specific levels on different attributes. Those levels are 
used in dominance structuring using positive or negative evaluations relative 
to the anchors. 

A concrete reference alternative may also be chosen among the alterna¬ 
tives attainable and present in the decision situation. The reference alterna¬ 
tive may later serve the function of a measuring tool for sorting the 
remaining alternatives into possibly preferred and unacceptable options. 
The importance of a reference alternative was illustrated by Tyszka (1983) 
and Payne (1982), who showed that among two equally valued alternatives, 
introduction of a third reference alternative that was dominated by just one 
alternative of the first pair resulted in the choice of the nondominated alter¬ 
native. A promising alternative (Montgomery 1983; Montgomery & Sven- 
son 1989b) may be adopted following scanning among the immediately 
available alternatives, or it may be created and after that incorporated into 
the set of decision alternatives. 

The elimination of nonacceptable alternatives is a cognitively important 
process because it reduces the set of alternatives and saves cognitive effort so 
that energetic resources may be allocated to the processing of the remaining 
alternatives. To exemplify, the use of the conjunctive (Svenson 1979) or 
satisficing rule (Simon 1955) leads to the rejection of all alternatives with 
aspects below certain criteria on the different attributes. Elimination may 
also be performed through, for example, a disjunctive decision rule, which 
prescribes that an alternative is selected (to the pool of alternatives to be 
considered further) by a single valuable aspect exceeding a criterion value on 
that attribute. Alternatives not selected in this way are not considered any 
further in the decision process. 

If all alternatives are eliminated, then reappraisal of values and decision 
alternatives must follow if the decision maker is ever going to make a deci¬ 
sion. If there is only one alternative left, this alternative will be the final 
choice (exit to the left in Figure 3.1). If two or more alternatives remain, 
comparison follows. Although more than two alternatives may remain in 
this stage, most of the cognitive effort is often concentrated on the promis¬ 
ing alternative and its most serious competitor (Montgomery & Svenson 
1989b; Dahlstrand & Montgomery 1989). 

Completion and/or restructuring of the mental representation is done in 
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response to the requirements arising from a misfit between the goal of a 
decision process (to find a dominating alternative, an alternative that may 
be justified for oneself and others* an alternative that fulfills just one crite¬ 
rion such as being the cheapest, an alternative most similar to what a signifi¬ 
cant other would choose, an alternative with the greatest emotional appeal, 
an alternative favored by as many arguments as possible, and so forth) and 
the actual situation. Montgomery (1983) exemplifies restructuring with 
dominance structuring, and completion of information is exemplified by 
Aschenbrenner, Albert, and Schmalhofer (1984), who found that informa¬ 
tion was searched until the summed cumulative difference in attractiveness 
between two alternatives exceeded a given number. Notice that reappraisals 
of the value system leading to restructuring of facts and of mappings of 
values in conjunction with information search may be quite urgent at this 
point in order to arrive at a final decision. 

When a mental representation has been achieved, which permits the appli¬ 
cation of one or more effective decision rules, the preferred alternative may 
finally be selected and the decision announced and implemented. It is inter¬ 
esting to note that the preferred alternative sometimes may be predicted 
quite early in the decision process (Montgomery & Svenson 1989b), which 
means that the Initially promising alternative has a higher chance of being 
finally selected than its competitors. 

Postdecisional processes have Interested few researchers, and yet it is im¬ 
portant to know about them. In fact, most of our decisions about the future 
are discussed or investigated postdecisional processes in terms of changing 
values (Festinger 1964; Janis & Mann 1977) and memories of facts (Fisch- 
hoff 1975). 

CONCLUDING REMARKS 

Decision aiding should heighten the sensitivity of decision makers, en¬ 
abling them to detect in time when there is a decision problem. A decision 
maker monitors a continuous stream of events and actions, most of which 
seem to follow naturally and almost automatically from past decisions. 
However, in a dynamic world, it is important to realize when a new decision 
has to be made. Otherwise, changing conditions and external forces will 
gradually start making decisions by themselves. If a decision problem is not 
detected In time, reversals of already started developments may have to be 
part of a decision when It is finally made. A decision maker who is late in 
detecting such creeping developments faces fewer but more difficult decision 
problems than the more alert decision maker. Relating to the framework of 
this chapter, detection of levels 1 and 2 decisions in need of processing at 
higher levels should be part of any decision-aiding technique. Decision aid¬ 
ing is made for the future. Therefore, it Is important to remember that such 
techniques should prepare the decision maker for future developments be- 
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yond the immediately perceived consequences and inform him or her about 
signs in the future that may appear, indicating the need for a new decision. 

The creation of decision alternatives may be hampered by an initially 
promising option which has been shown to have a psychological advantage 
over its competitors. One of the powers of decision-aiding tools is to keep 
alternatives eliminated early in the process alive for later reappraisal, which 
may not have happened in unaided decision making. 

In decision aiding, it is important to elicit factual scenarios of what might 
follow different decisions, but it is also important to allow for and expose 
the uncertainty of values, which is inherent in all decision making (Fisch- 
hoff, Slovic & Lichtenstein 1980). This concerns not only conflicting goals 
derived from the same value system. It also refers to the fact that one does 
not always know what values apply on level 3 and 4 decision problems. As 
illustrated by the dominance search model, decision makers actively use 
their own uncertainty of values in the restructuring of a problem in order to 
facilitate a decision. This is partly reflected in the decision-analysis method 
presented by Moskowitz, Wong and Chu (1989). 

Decision-aiding techniques often provide a structuring of a decision prob¬ 
lem in such a way that computer algorithms (after necessary parameter se¬ 
lection) may be applied in solving the problem. The human decision maker 
has a more flexible approach to decision making than a computer program. 
The human decision maker uses different decision rules in different situa¬ 
tions. In order to approach the human decision process and thereby gain 
increased credibility, decision-aiding techniques might profit from incorpo¬ 
rating a wider set of decision rules. In addition, a relatively limited set of 
alternative decision rules may improve the trustworthiness of decision-aid¬ 
ing techniques significantly. Such decision-aiding techniques would some¬ 
times violate one of the trademarks of decision-aiding tools, consistency. 
This is because some decision rules, all reasonable in isolation, give incon¬ 
sistent results when applied to the same set of alternatives. It would then be 
up to the decision maker to choose, on a meta level, the representation of a 
decision problem and the decision rule or rules he or she finds most applica¬ 
ble in each situation. Still, a good decision aid would offer information 
about inconsistencies to the decision maker and provide advice about how to 
solve them through proper representation of the decision problem and appli¬ 
cation of a consistent set of decision rules. 
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Part Two 


Approaches Based on Management 
Science/Operations Research 

Like any other form of decision-aiding software, management science/ 
operations research (MS/OR) packages involve (1) goals to be achieved, 
(2) alternatives available for achieving them, and (3) relations between 
goals and alternatives in order to arrive at (4) tentative conclusions as to 
which alternative is best. On each of those four elements, MS/OR pack¬ 
ages tend to be distinctive from other packages, especially from those 
oriented toward spreadsheet-based multicriteria decision-making 
(MCDM) packages. 

The typical MS/OR package involves a single goal to be achieved. 
The single goal may be arrived at by aggregating other subgoals, often 
by imposing a common measurement unit like dollars so the subgoals 
can be added together. The alternatives tend to be dichotomies of a go/ 
no-go variety (drawing upon the logic of finite mathematics), or they 
involve a continuum of alternatives (drawing upon the logic of calculus). 

The most common MS/OR packages deal with payoff matrices, deci¬ 
sion trees, linear programming, and optimum-level curves in accordance 
with the most common methods discussed in MS/OR textbooks. A pay¬ 
off matrix typically involves a dichotomous choice on the two rows, two 
possible conflicting events or states of nature on the columns, and an 
indication in the cells of what the payoffs are likely to be to each alterna¬ 
tive with the occurrence of each event. This can be contrasted with an 
MCDM table where the rows consist of a variety of alternatives. There 
are usually about four in a public policy problem: conservative, liberal, 
neutral, and mutually optimizing alternatives. An MCDM table also 
tends to show goals rather than probabilistic events on the columns, and 
the cells indicate how conducive or adverse the alternatives are to each 
goal. 

Another typical MS/OR format is the decision tree, usually pictured 
as looking like a tree on its side, with branches and subbranches. The 
branches generally represent alternative possibilities that depend on the 
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occurrence or nonoccurrence of probabilistic events. Such a way of de¬ 
picting a decision-making problem has aesthetic appeal. Unfortunately, 
that format may have considerable difficulty dealing with multidimen- 
sionalixy, missing information, multiple alternatives, conflicting re¬ 
straints, and simplicity, especially in seeking to draw a variety of 
conclusions through trial-and-error experimentation. 

A third type of MS/OR software is linear programming. Unlike pay¬ 
off matrices and decision trees, which emphasize discrete alternatives, 
linear prog rammi ng specializes in allocating scarce resources along a 
continuum of possibilities to a set of variables under various kinds of 
quantitative constraints. The scarce resources may be money, time, or 
people. The variables may be activities, places, tasks, or other objects. 
In terms of form rather than function, linear programming involves 
maximizing or minimizing an objective function or algebraic expression 
subject to constraints generally in the form of inequalities like greater 
than or less than. As with other MS/OR methods, it sounds and looks 
sophisticated. In reality; it, in effect, fails to recognize that the world is 
round and that increased allocations of scarce resources to any budget 
category or goal tends to produce increased benefits but at a diminishing 
or nonlinear rate. If one attempts to switch to nonlinear versions of lin¬ 
ear programming, then one runs the risk of greatly increased complexity 
and possibly decreased validity as the solution gets stuck on a local peak 
or in a local hole still looking for the global solution. 

A fourth type of MS/OR software involves optimum-level curves. 
This kind of software is more explicitly oriented toward calculus solu¬ 
tions than other MS OR software. In theory, it could be especially rele¬ 
vant to finding an optimum level where doing too much or too little is 
undesirable. The methodology assumes that one can determine an equa¬ 
tion showing the relation between varying amounts of a policy input and 
varying degrees of goal achievement. This is supposed to result in some 
curves showing a positive nonlinear relation between the policy and vari¬ 
ous benefits and some curves showing a negative nonlinear relation. By 
summing the curves, one is supposed to get an overall hill-shaped or 
valley-shaped curve. The top or bottom indicates the optimum level. 
Such a system is not so capable of dealing with multiple policies, multi¬ 
ple measurement, or problems in which the relations between the poli¬ 
cies and goals tend to be reciprocal or determined by outside factors. 
Such relations may be especially present in public policy problems, 
where the social indicators (like crime) have more influence on how 
much is allocated to various policies (like prisons) than the policies have 
on the sexual indicators. 

Although MS/OR models may have drawbacks, they also have some 
advantages over competing MCDM or expert systems models. Decision 
trees and optimum-level curves, for example, can communicate ideas as 
visual aids better than a set of expert branching rules or a matrix of 
alternatives and goals, although that is partly a matter of taste. MS/OR 
models may work fine with industrial engineering problems where one 
can obtain the data needed to satisfy the models and where the assump- 
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tions are realistic, such as linear and nonreciprocal relations. Some MS/ 
OR packages are better than others. One purpose of Part Two is to make 
comparisons within MS/OR models, as contrasted to the more difficult 
problem of comparing MS/OR, MCDM, and expert systems models. 
The MCDM and expert systems models are discussed in Parts Three and 
Four. 
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A Comparison of Linear 
Programming Software 
on Microcomputers 


Ramesh Sharda 


One of the most powerful decision-aiding techniques, which has witnessed 
tremendous applications and software development, is linear programming 
(LP). While many other techniques covered in this book are appropriate for 
decision making, LP is especially important due to its versatility. No other 
technique has received as much attention as LP in algorithmic development, 
with the result that very large LP problems can now be solved. As a conse¬ 
quence, LP has been used not only in problems in which linearity is present 
but also as an intermediary tool where its direct use would not be possible. 
For example, LP can be used in nonlinear problems where piecewise approx¬ 
imation of the nonlinear functions yields large LP problems. Versions of 
simplex method have been developed where multiple criteria are present. 
Thus, LP has been employed as a computing tool where the original prob¬ 
lem does not appear to be initially appropriate for linear programming. 
Thoughtful reformulations can allow LP to be a powerful decision-aid tech¬ 
nique. Recent developments in LP software make it ever more attractive. 

While LP software for microcomputers has been available ever since the 
micros became popular, recently there has been a major growth in the power 
and capability of such software. LP software has now been developed to 
take advantage of powerful microcomputers (such as 80386-based machines) 
and certain sophisticated features to enable a decision maker to model, man¬ 
age, solve, and analyze LP problems more efficiently. 

This chapter provides an up-to-date compendium of new packages that 
solve LPs and reports on the user-interface, user-control, and mathematical 
programming capabilities of this software. My evaluation also focuses on 
the performance of each package when solving a set of test problems. Over 
forty medium to large LPs were solved. On the basis of functional perfor- 
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mance and other characteristics (such as ease of use), I identify standout 
systems, make recommendations for product enhancement, and comment 
on future directions in developing new software. 


LINEAR PROGRAMMING SOFTWARE 

Over two dozen LP packages are currently available for MS-DOS com¬ 
puters. Since a recent survey paper (Sharda 1988) has listed the information 
on availability of these programs, this chapter includes only a list of soft¬ 
ware introduced or updated since the survey paper was published. (Refer to 
Sharda 1988 for several other LP programs.) Table 4.1 lists product names, 
publishers, maximum problem size as stated by the publishers, and prices 
for commercial customers. Most vendors offer site licenses, significant edu¬ 
cational discounts, and smaller versions at lower prices. MINOS and XMP 
are mainframe programs that have been ported to MS-DOS computers. 
Their developers note that these programs are still in developmental stage to 
lake advantage of personal computer (PC) architecture and interface. 


User Interface Issues 

The major enhancement in LP software has occurred in user interface. 
Most of the LP software available today is capable of reading LP problems 
in the Mathematical Programming System (MPS) format. This capability is 
important for moving LP problems previously solved on a mainframe to a 
PC. The MPS format also allows for a standard format to exchange prob¬ 
lems for different LP systems. Table 4.2 identifies programs capable of read¬ 
ing MPS files. LINDO, XA, MPSX-PC, and XPRESS-LP are also able to 
save an MPS format file for an LP problem developed using other formats. 

Perhaps the most significant recent achievement of LP software vendors 
has been to develop spreadsheet-compatible programs. Spreadsheet pro¬ 
grams have become almost integral parts of microcomputers. Most PC users 
are familiar with spreadsheets. While this familiarity does not help improve 
the analytical aspects of formulating an LP problem, the convenient data 
entry* and editing features of spreadsheets make it easy to develop the LP 
matrix. Spreadsheet programs allow blocks of numbers or relationships to 
be copied from one location to another. This feature can be useful in devel¬ 
oping LP problems that have blocks of similar structure. Report generation 
and graphics features of spreadsheets can be used for developing final, pre¬ 
sentable reports quickly from the optimal solution stored in a spreadsheet 
file. These features have resulted in the popularity of spreadsheet-compati¬ 
ble LP software. This is the only class of LP software that has received wide 
coverage in popular media. For example. What’s Best! a spreadsheet-based 
LP package, has been mentioned in the Wall Street Journal , and New York 
Times. It also received PC Magazine’s Technical Excellence award. 

Table 4.2 lists the LP programs compatible with spreadsheets. There are 
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various degrees of this compatibility. MSLP-PC offers only a spreadsheet¬ 
like input/data editor. LPS-867, MPSX-PC, and XPRESS-LP are able to 
read a problem as well and write the solution to a spreadsheet file. VINO, 
What’s Best! and XA offer the most complete integration with a spread¬ 
sheet. These programs can be used from within the spreadsheet program 
such as Lotus 1-2-3. The user creates an LP model in a spreadsheet file, 
presses one or two keys to invoke the optimizer, and solves the problem as if 
some cells were being recalculated. This makes it appear that the spreadsheet 
program is endowed with an optimizer. If spreadsheet compatibility is a ma¬ 
jor concern, these programs deserve a careful look. What’s Best! can also be 
purchased with spreadsheet templates for some common LP applications. 

As the capability of LP software to solve medium to large problems 
evolved, it became apparent that such problems would perhaps be developed 
using matrix generators and modeling languages, similar to the mainframe 
environment. Most LP problems have only a few types of constraints. Using 
a modeling language, one needs to enter those equations just once. Similar 
constraints are automatically generated using only the data defined by the 
user. Many advances have been made in this area. GAMS, a popular model¬ 
ing system on the mainframes, has been ported to PC and combined with 
MINOS to offer a complete modeling and analysis package. GAMS now 
also includes XMP and Zoom for linear and integer programming prob¬ 
lems. FORTLP will be available with a menu-based model development sys¬ 
tem, CAMPS. Developers of LINDO packages have announced a modeling 
language, LINGO, for use with their software. An OMNI-like matrix gener¬ 
ator will be available for MPSX-PC. XPRESS-LP comes with a fairly com¬ 
plete modeling language. 

MPL, a new entry in mathematical programming languages, is not an LP 
package in itself. Rather, it allows an analyst to build a model using its lan¬ 
guage and then save the model in several commonly used formats, such as 
those accepted by XA and LINDO. It can also save the problem in MPS 
format so that the problem is acceptable to virtually any LP package. This 
allows one to use MPL as a front end to one’s own favorite LP software. 

Some packages that do not offer a modeling language have an expressions 
format available to enter the complete LP problem just as one would write it 
out. LINDO packages, MILP, MPSX-PC, and XA offer this capability. The 
options available in the expressions format vary among packages, with 
MPSX-PC and XA being the most general. This format is quite appropriate 
for small LP problems. As the problem size increases, one realistically has to 
use a file generated in MPS format through an external program, a spread¬ 
sheet file, or a modeling language. 


Performance Issues 

First, we note several programs offering mixed integer programming capa¬ 
bility. Two additions to the list of programs with MIP option are XMP and 



Table 4.1 

General Information on LP Software 


Software 

Publisher 

Size of 

Price Problem 

Remarks 

FORTLP 

Dr. G. Mitra * m<2,000 

Department of Mathematics n<3,000 

and Statistics, Brunei University NZ<20,000 

Uxbridge, Middlesex 

Great Britain UB8 3PH 

Requires a 

FORTRAN compiler. 

GAMS/MINOS 

GAMS Dev. Corporation 
2828 Albemarle STNW 
Washington, D.C. 20008 
(202) 232-5662 

$1,600 Depends 
upon the 
system 
configur¬ 
ation 

Also marketed by 

The Scientific Press 
651 Gateway Blvd. 

S. San Francisco, CA 
94080-7014, 

(415) 583-8840. 

HYPER UNDO 

UNDO Systems, Inc 

1415 N. Dayton St. 

Chicago, IL 60622 
(312) 871-2524 

$1,500 m< 1,999 
n<3,999 
NZ<32,000 
I<200 

Smaller versions 
available for lower 
prices. Marketed by 
the Scientific Press. 

UNDO/386 


$2,500 m< 5,000 
n< 15,000 

NZ<100,000 

LPS-867 

Applied Automated 
Engineering Corp. 

65 S. Main Su Bldg. B 
Pennington, NJ 08534 
(609) 737-6800 

$500 m< 2,500 
n<5,000-m 
NZ< 16,000 

Smaller versions 
available for 
lower prices. 

M1LP 

Akademiai Kiado 

Hungarian Academy of Sci. 
Varoshaz U. L, H1052 
Budapest, Hungaiy 

$2,500 m<800 
n<2,000 
NZ<32,000 


MTNOS-LP 

Systems Optimization * 

Laboratory 

Department of Operations Research 

Stanford University 

Stanford, CA 94305 


MPL 

Maximal Software 

Klapparas II 

IS-110 Reykjavik 

Iceland (354-1) 78243 

* 

A Mathematical 
Modeling Language 
Compatible with 
many LP systems. 

MSLP-PC 

CompuED 

PO Box 35544, Sul E. 
Vancouver, B.C, 

$80 m<200 

n<400 

smaller version 
available for $20. 


Canada, V6M 4G8 
(604) 224-6228 
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Table 4.1 (continued) 


Size of 


Software 

Publisher 

Price 

Problem 

Remarks 

PC-Prog 

QMS 

P.O. Box 12597 

1110 AN Amsterdam Z-O 
The Netherlands 
31-20-865640 

* 

m<250 

n<500 

Includes quadratic 
and integer 
programming. 

VINO 

See UNDO 

$695 

m<2,000 

n<4,000 

NZ<32,000 

I<200 

Smaller version 
available at a 
lower price. Also 
sold by Scientific 

Press. 

What’s Best 

General Optimization, Inc. 
2251 N. Geneva Terrace 
Chicago, IL 60614 
(800) 441-BEST 

$995 

m< 1,600 
n<4,000 
NZ<32,000 
I<400 

Smaller versions 
are available at 
lower prices. Also 
sold by Holden-Day 
Inc. 4432 Telegraph 
Ave., Oakland, CA 
94609, (415) 428-9400. 

XA 

Sunset Software Technology 
1613 Chelsea Road, 

Suite 153 

San Marino, CA 91108 
(818) 284-4763 

$1,195 

m< 6,000 

n<26,000-m 

NZ<32,000 

This version requires 
80386. Other versions 
for 8088/80286 
machines are available 
with somewhat 
reduced size 
capabilities. 

XMP 

XMP Software, Inc. 

930 Tahoe Blvd., #802-279 
Incline Village, NV 89451 
(702) 831-4967 




XPRESS-LP 

Dash Associates £1,500 

Blisworth House, Church Lane 
Blisworth 

m< 1,000 
n<2,000 

Smaller version 
available for a 
lower price. 


Northants NN7 3BX £875 

Great Britain 
(604) 858993 

Note: m = number of rows, n = number of columns, NZ = number of non-zero elements, 
E, L, G = number of constraints of each type, I = number of integer variables, 
* = Available for a small (undetermined) fee. 
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Table 4.2 

Functional Features of LP Software 
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XPRESS-LP. Sharda (1988) summarizes the numerical limits on integer var¬ 
iables for various programs. 

I tested the performance of LP software available to me by solving a num¬ 
ber of test problems. Since the test problems were available in MPS format, 
I tested only those packages that could read MPS files. FORTLP and 
LINDO/386 were not available in time for the tests reported in this chapter. 
Further, the publishers of MINOS and XMP stated that the PC versions of 
their codes were running but not optimized for use on a PC yet and re¬ 
quested that these not be included in the evaluation. 

The test set consisted of forty-nine problems collected from a number of 
sources. Most of these problems are described in Gay (1985). FARM, 
HEDGE, REFINE, and WILLETT are described in Sharda and Somarajan 
(1986). These problems range in size from 56 nonzero coefficients to 14,706 
nonzeroes. Most of these problems have been accepted as fairly standard as 
a test set, even for testing mainframe LP codes. 

The problems were solved using an IBM PS/2 model ’80 computer using 
Intel 80386/16 MHZ processor and 80387 math coprocessor. This machine 
has a 44 megabyte hard disk, and 2MB RAM, running DOS 3.3. RAM disk 
was used wherever possible. 

Table 4.3 gives the times for each package to solve problems to within 
1 percent of the solution identified by MPSX/370. Errors are marked where 
a package could not solve a problem. There are a number of possible errors. 
First, some problems may be too large for a package. This may be clear 
from the product literature. Occasionally, the initial problem size is accept¬ 
able, but the files may become too large for a package as the interactions 
progress. A second category of errors is due to a package’s limitation on 
accepting MPS files. Our test problems employ fairly general versions of 
MPS format accepted by MPSX/370, but PC packages occasionally are un¬ 
able to read them. For example, a program may not accept coefficients in 
scientific notation, it may not accept RANGES, it may not accept 
BOUNDS, or it may not accept identical row and column names. These er¬ 
rors can usually be fixed by the user. A third category of errors, rather seri¬ 
ous, is In accuracy. A package may solve a problem to an incorrect optimal 
solution, or it may declare it unbounded or infeasible. Finally, some prob¬ 
lems were perhaps so large that the programs got stuck or did not lead to an 
optimal solution in over nineteen hours, the time limit that a user will find 
acceptable. 

Table 4.3 gives the times for a package to solve each problem and also the 
various errors. It indicates that no PC package was able to solve all of the 
forty-nine problems. XA solved the maximum number of problems (forty- 
six), with accuracy or timing errors on the other three problems. LPS-867 
solved forty-two of the problems. XPRESS-LP solved forty-one problems 
with no accuracy error and six unacceptable format errors. When the format 
errors were fixed, it was able to solve these six problems as well. This means 


Table 4.3 

Test Statistics on a 80386 Machine (in seconds) 


-32E. 

NO. OF 

NO. OF 

i NON 

UNDO 

MI IP 

XA 

XPRES5 

LF5 

OAKLAND 

SAKE 

SONS 

EELS 

ZEROS 




LP 

347 


I5FV47 

gnn 

2393 

11949 

r. 

r 

4117? 

7404.95 

A 


Wl». s tLL 

ej 

15* 

jr*sn 

12.48 

9 

9 

5.71 

10 

14 

1 

•> c IRi3 

28 

s0 

lie 

0.5a 

0.27 

1 

2.03 

i 

BANS* 

EOc- 

778 

2935 

1444.4 

352 

210 

151.09 

433 

D 

BAEESEIM 

43 

108 

257 

1.31 

0.41 

1 

1.48 

1 

c 

BEAiQNFD 

174 

434 

3450 

39.57 

10 

9 

13.18 

112 

82 

stHii 

171 

*74 

1073 

32.19 

14.48 

21 

15.04 

23 

92 

BERBER 

45 

198 

480 

2.85 

1.59 

4 

14.45 

4 

a 

sOxEwB 

234 

5*9 

1759 

750.73 

29 

74 

B 

c4 

112 

BRANDY 

221 

470 

2371 

Mil. 79 

115.42 

158 

54.3 

159 

c 

CAPRI 

272 

425 

2058 

158.14 

108 

101 

123.44 

1055 

B 

DINAKICO 

417 

944 

4394 

0 

2857.9 

434 

172.08 

1075 

D 

i226 

E7ANACR0 

224 

401 

508 

1089 

3244 

2890 

82 

421 

33.44 

181.04 

58 

280 

24.22 

129.84 

S 

349 

232 

C 

r ARfl 

S 

20 

54 

0.34 

0.05 

0 

1.75 

0 

0 

FFFFF80O 

525 

1379 

4740 

409.89 

C 

235 

301.87 

5592 


’-IIHAR 

325 

TT? 

2924 

438.52 

94.04 

133 

71.84 

1122 

A 

FORESTRY 

402 

1005 

419a 

1383 

470 

510 

327.3 

3342 


FORPLAN 

142 

5S3 

5073 

3525 

121 

49 

54.98 

434 


SrxhPXC 

417 

1092 

3447 

213.01 

204 

352 

158.4 

442 


cSOHlS 

mm 

301 

441 

5 4e 

1387 

594a 

8759 

3513 

A 

C 

C 

503 

1184 

484.97 

C 

B 

A 

A 

SRDI7 

141 

*42 

2774 

433.2* 

548 

90 

45.58 

597 

c 

HEBSE 

93 

250 

480 

5.39 

5 

4 

a.l 

53 

16 

ISRAEL 

175 

317 

2533 

111.05 

57 

75 

77.17 

92 

173 

MEEN 

443 

3584 

14451 

950a 

A 

5203 

B 

A 


PILBT4 

411 

1411 

5554 

A 

C 

C 

2454.14 

c 


PILQT.JA 

941 

1958 

1470c 

A 

A 

D 

5 

A 


=R0B12B 

381 

492 

3373 

299.19 

202 

192 

84.95 

1413 

1324 

PR6B31B 

384 

941 

4448 

208.57 

120 

219 

102.27 

1203 

631 

PR0B43B 

448 

1253 

5348 

253.84 

182 

295 

128.41 

1917 

RECIPE 

92 

272 

844 

2.08 

1.24 

3 

4.06 

4 

1 

13 

REFINE 

30 

a3 

185 

0.23 

1 

I 

2.24 

l 

SC205 

204 

409 

758 

40.52 

50 

45 

2^. .zo 

48 


SEASR7 

130 

270 

453 

30.1 

11 

9 

12.41 

173 

c 

scorpion 

389 

747 

2097 

C 

71 

57 

44.02 

123 

484 

SCES8 

491 

1440 

4520 

531.22 

507 

511 

235.41 

5135 

5 

5C551 

78 

838 

3224 

73 

7? 

47 

23.95 

32 

IBS 

SESB6 

148 

1498 

5814 

241 

171 

187 

91.94 

952 


SETAPI 

311 

791 

3483 

75.02 

47 

113 

59.43 

725 

A 

SCTAP2 

1101 

2931 

14395 

527.54 

A 

C 

535.38 

1497 


SH&RE1B 

118 

343 

1300 

125 

47 

54 

31.53 

57 

266 

SHELL 

537 

2312 

5437 

211.35 

144 

255 

151.21 

2040 

D 

SBIP04S 

403 

1841 

4213 

85.29 

101 

274 

104.43 

148 


STAIR 

357 

324 

4214 

C 

c 

384 

B 

148B 


5TAK0ATA 

340 

1435 

3398 

10.4 

12 

97 

B 

25 


STAKBHPS 

*48 

1543 

4154 

102.22 

42 

142 

B 

75 

B 

V7P.BASE 

19? 

402 

1113 

48.22 

54 

42 

B 

118 


MILLET! 

185 

479 

2532 1 

327.94 

C 

251 

131.11 

2115 

A 


i PSOBiBS SOLVE# CORRECTLY 42 3V w 4J (7 

.\ote: A = limitation of problem size, B = limitation of input, C = limitation of accuracy 
D = package gets stuck (takes too long). 






Linear Programming Software on Microcomputers 


103 


that XPRESS-LP would be able to solve forty-eight of the forty-nine prob¬ 
lems. However, Table 4.3 indicates an input format error for XPRESS-LP 
for these six problems since the other packages were able to accept those 
problems. 

LPS-867, LINDO, and MILP were also able to solve most of the prob¬ 
lems, although they had more accuracy of precision errors. 

To compare these programs on the basis of speed, I used the pairwise 
comparison approach suggested by Lootsma (1980). The pairwise rank of 
package k with respect to package /, r kl is computed as 


T = &T ik /M kl )/C(k,[) 
kl &T n /M kl yc(k 3 r) 
where 

T ik = (Min k t ik )/ t ik is a minimum time ratio, 
t ik = running time of package k on problem /, 

C(k,l) = cross-section of successfully solved problems by packages k 
and /, and 

M kl = number of problems successfully solved by both packages. 

Table 4.4 shows the pairwise comparison of speed of these LP programs. 
It shows that XPRESS-LP has a higher rank in speed compared to any of 
the other packages, based on these test problems. MILP ranks higher than 
LINDO, LPS-867, and XA. XA ranks higher than LINDO and LPS-867. 

These results show that there are some differences in performance of a 
package in terms of speed. However, all of the times are in an acceptable 


Table 4.4 

Pairwise Comparison Matrix of Selected LP Packages 



LINDO 

MILP 

XA 

XPRESS 

LP 

LPS 

867 

LINDO 


0.64 

0.73 

0.55 

1.47 



(34) 

(37) 

(34) 

(36) 

MILP 

1.56 


1.16 

0.80 

2.15 


(34) 


(39) 

(35) 

(38) 

XA 

1.37 

0.86 


0.73 

2.01 


(37) 

(39) 


(39) 

(41) 

XPRESS-LP 

1.82 

1.25 

1.37 


2.81 


(34) 

(35) 

(38) 


(37) 

LPS-867 

0.68 

0.46 

0.50 

0.36 



(36) 

(38) 

(41) 

(37) 



Note: The numbers in parentheses indicate the size of the common cross-section between pairs 
of codes. 
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range for use on a PC, and thus other features of a program should be care¬ 
fully weighed in selecting an LP optimizer. VINO and What’s Best! use 
LINDO as the internal optimization tool; their speed performance can be 
expected to be similar to LINDO’s. These comparisons are based on the 
software versions available as of October 1988. New versions are constantly 
being developed by LP software vendors. 

FUTURE DIRECTIONS 

Added Hardware Support 

Some vendors have attempted to reduce the time to solve large problems 
by supporting third-party hardware designed for improving arithmetic oper¬ 
ations. For example, LINDO/386 can be run using a Weitek floating point 
coprocessor. A sample problem with 516 rows and 1,035 columns was solved 
on a Compaq 386/25 MHZ machine using LINDO/386 and Weitek copro¬ 
cessor in 9 seconds. The same problem took seven minutes, fifty seconds on 
a PC/XT with 8087 math coprocessor. LPS-867 has been adapted to solve 
problems using Definicon Systems boards, with a twenty-fold improvement 
in speed on 80286 computer as compared to a PC/XT. The 80386 version of 
XA is supposedly able to solve problems 50 to 500 percent faster than earlier 
versions. XPRESS-LP is being modified to work with parallel processors on 
microcomputers. Thus, it is clear that LP software developers are moving to 
take advantage of hardware enhancements. More work in this area will likely 
lead to mainframe performance on micros in a short time. 


User Interfaces 

More developments are expected in modeling languages. Efforts are un¬ 
derway to generate an LP problem automatically using expert systems and 
natural language processing technology. Intelligent interpretations of solu¬ 
tions produced by optimizers are also being developed (Greenberg 1987). We 
can also expect windows-based problem development interfaces as the new 
operating environments become more popular. A related growth field will be 
graphics-based model development systems. 


Domain-Specific Models 

Microcomputers offer the potential for extending the benefits of opera¬ 
tions research modeling to a much larger audience than was possible with 
mainframes. This will be facilitated by providing templates of LP models 
formulated for specific problems. What’s Best! is available with templates 
for several common LP applications. Using hypertext technology, this can 
be done for many other applications. 
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It is clear that the LP software is constantly improving to exploit the ad¬ 
vances in microcomputers. We are already able to solve problems with sev¬ 
eral thousand nonzeroes on a PC. The forthcoming advances in user 
interfaces and model development systems are likely to make LP systems 
even more accessible for decision making. 
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DECISION ANALYSIS-BASED DECISION SUPPORT 

Decision analysis (Keeney 1982; Raiffa 1970; Samson 1988; von Winter- 
feldt & Edwards 1986), the application of decision theory to real-world 
problems, has been referred to as the oldest, best established, and most ar¬ 
ticulate school of decision support systems modeling (Stabell 1987). Deci¬ 
sion analysis focuses not on modeling a system but on modeling the decision 
(von Winterfeldt 1988). It provides methods for decomposing problems into 
sequences of choices, outcomes, events and their associated probabilities, 
and estimates of the preferences of decision makers for outcomes. The 
founding principle is that humans can process fragmented knowledge in 
the other forms reliably but have difficulty dealing with the aggregation of 
the fragments as complex problems (Leal & Pearl 1977). 

Decision analysis has been successfully and extensively applied to less 
well-structured problems (see Corner & Kirkwood 1991 for a recent survey 
of decision analysis applications). Keeney (1982) and Ulvila and Brown 
(1982) list a number of cases, as well as a number of large organizations 
that, by establishing groups within the organization, have committed them¬ 
selves to the ongoing application of decision analysis. The many published 
cases include public utility technology choice (Keeney, Lathrop & Sicherman 
1986), nuclear power plant site selection (Kirkwood 1982), information tech¬ 
nology strategy (Brooks & Kirkwood 1988; Williams 1986), new product de¬ 
velopment (Conway 1986), manufacturing strategy (Samson 1987), and 
general insurance strategy (Samson 1986). These cases and others provide 
evidence of the wide applicability of decision analysis to decision support. 

A major barrier to the further expansion of the use of decision analysis is 
the level of expertise needed to carry it out; many decision makers require 
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intermediaries between them and the computing and modeling techniques to 
solve their problems. An important part of the expertise of the decision ana¬ 
lyst is the art of structuring problems. This chapter proposes a method for 
structuring problems based on influence diagrams that will enable more de¬ 
cision makers to avail themselves of the power of decision analysis. 


REPRESENTING DECISION PROBLEMS AS 
INFLUENCE DIAGRAMS 

Decision analysis problems can be represented by decision trees; however, 
influence diagrams (Agogino & Rege 1987; Barlow 1988; Shachter 1988; 
1986), a more recent representation, facilitate the development of decision 
models. Influence diagrams are Bayesian belief networks (Pearl 1988), 
which include decision and outcome nodes. They have already been applied 
in problems relating to construction risk management (Ashley, Stokes & 
Perng 1988), environmental management (Varis & Kettunen 1988), milling 
plant management (Agogino, Guha & Russell 1988), petroleum geology 
(Reid et ah 1985), forensic science (Lou & Gammerman 1990), health tech¬ 
nology assessment (Shachter, Eddy & Hasselblad 1988), and therapy plan¬ 
ning (Quaglini et al. 1989). 

Influence diagrams are a network of choice, event, and outcome nodes 
(Figure 5.1). Choice nodes are represented as rectangles, event nodes as cir¬ 
cles, and outcome nodes as diamonds. The choice nodes describe decisions. 
Inside decision nodes are the alternatives that may be selected by a decision 
maker. Event nodes can represent endogenous and exogenous events and 
states that are probabilistic or deterministic. Probabilistic or chance events 
are described Internally by probability distributions. Deterministic events 
represent variables that can be determined precisely once their predecessors 
are known. Outcome nodes represent goal states and contain a function 
used to measure selected policies. Links into events show other events and 
decisions that condition that event. Links into decision nodes show informa¬ 
tion that is knowm when that decision is made. 

An influence diagram can be viewed from three levels: the relational level, 
the numerical level, and the functional level (Moore & Agogino 1987). The 
labeled nodes and arcs describe a problem at the relational level, showing 
the major problem components and their Interdependence. Once built at the 
relational level, the influence diagram can be specified at the numerical 
level. This can be seen as specifying the internal values of nodes, the alterna¬ 
tives in choice nodes, the probability distributions in probabilistic events, 
and the functions in deterministic events and the outcome node. A numeri¬ 
cally specified influence diagram can be solved to produce a solution in the 
form of the set of decisions that maximize expected utility. 
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Figure 5.1 

Insurance Planning Model 



SOLVING INFLUENCE DIAGRAMS 

Horvitz, Breese, and Henrion (1988) have described a number of ap¬ 
proaches to solving influence diagrams and propagating belief through 
Bayesian belief networks: brute force methods, exact methods, and stochas¬ 
tic simulation. Brute force methods calculate the joint distribution for a par¬ 
ticular variable, from which marginal probabilities for any value can be 
obtained by summing over the relevant values in the joint distribution. The 
difficulty with this approach is that the computational task increases expo¬ 
nentially as the number of variables increases, and the method is impractical 
for more than a few variables. 

Exact methods exploit the topology of networks. They are restricted to 
particular topologies, analyze a particular diagram before selecting a least- 
cost solution strategy, or manipulate the topology of a diagram before solv¬ 
ing it. The best-known exact method for solving influence diagrams is the 
algorithm developed by Shachter (1986). Although exact approaches are less 
expensive in terms of solution time than brute force approaches, they have 
also been shown to be susceptible to NP hardness (Cooper, 1987). 

Stochastic simulation methods are typified by the iterative two-phase sim¬ 
ulation method that has been successfully applied in risk construction 
(Ashley, Stokes & Perng 1988). In the first phase, a variable distribution is 
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computed from its neighbors. The second phase samples a new value from 
the distribution computed in phase 1 and then reevaluates the distribution in 
the diagram. An advantage of this approach is that this procedure may be 
able to be implemented by carrying out some operations in parallel (Hor- 
vitz, Breese & Henrion 1988). 

Solution methods for influence diagrams are a matter of ongoing 
research. The traditional decision analysis approach to overcome com¬ 
putational difficulties is to keep problems small and tractable using atten¬ 
tion-focusing heuristics (Holtzman 1989). These heuristics keep factors that 
have the greatest impact on the solution in the problem while removing those 
that are insignificant. Algorithms such as Shachter’s (1986) combined with 
these heuristics provide a practical solution means for reasonably large 
problems expressed as influence diagrams. 

USING INFLUENCE DIAGRAMS TO 
STRUCTURE PROBLEMS 

Influence diagrams are superior to decision trees in their expressive power 
and their flexibility in structuring problems. An influence diagram can show 
the major concepts in a problem, separated from the values that variables 
representing these concepts may take. This allows influence diagrams to rep¬ 
resent large problems more concisely, enabling better communication for 
problem structuring and verification. Influence diagrams are used here as a 
grammar for communicating problem structure. 

Influence diagrams provide a medium with which the expert decision ana¬ 
lyst may be comfortable, as well as a means of communication between the 
analyst and the decision maker. They provide a mediating knowledge repre¬ 
sentation, a desirable feature of knowledge acquisition tools (Bradshaw et 
ah 1991). The following conversion process is supported: 

verbal description -► influence diagram solution. 

The ability to present models as high-level diagrams is a common and suc¬ 
cessful means of overcoming human cognitive limitations (Pearl & Verma 
1987). Graphic displays have been advocated as a means of external memory 
that is readily understood and accessed by users, and there is evidence that 
some users develop better understanding of a decision environment using 
such aids (Pracht & Courtney 1988). It is envisaged that the decision maker, 
with appropriate training and software support, will be able to develop deci¬ 
sion models working directly with influence diagrams. 

Howard (1989) has described a relevance diagram (an influence diagram 
without choice nodes) as a means to “get fragmented information out of 
people’s heads, onto paper, and ultimately into a computer.” The fragmenta¬ 
tion Into events allows appropriate experts to address only those parts of the 
problem where their expertise is applicable. Howard illustrates this in a rele- 
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vance diagram to estimate the price of oil in the year 2010. The marginal and 
conditional probabilities of relevant factors, such as scientific developments, 
political change, and exploration technology, are best assessed by appropri¬ 
ate and different experts. 

Decision trees have been grown from the root out by specifying a sequence 
of decision alternatives and events (Leal & Pearl 1977). Pearl, Leal, and Sa¬ 
leh (1982) have criticized this approach as not being compatible with the way 
decision makers structure problems. Further, the variables in a decision tree 
may be sequenced in more than one way. The tree used to assess probabilities 
is not the same as the one needed for decision making, and the assessment 
tree must be “flipped” to produce the decision tree (see, for example, Call & 
Miller 1990). Influence diagrams support different approaches to problem 
structuring while maintaining a model that represents the decision makers’ 
view of the problem that can be solved. 

Influence diagrams support the development of models in the more com¬ 
patible method of “forward-driven” action/event scenarios suggested by 
Pearl (1982). For complex problems, not all actions are known, and a “goal- 
driven” unraveling of a tightly woven cause-effect network is helpful. Keeney 
(1982) has also supported this approach to elicit more alternative choices 
when there are too few. Influence diagrams support a goal-driven approach 
in which the decision maker isolates relevant decisions and events and de¬ 
composes the problem by posing the question, “What variable can reduce 
the uncertainty about this variable?” (Owen 1984). 

The direction in which knowledge is elicited using influence diagrams can 
also change. For example, Shachter and Heckerman (1987) have suggested 
that sometimes the cause-effect direction may be a more appropriate means 
of capturing expert knowledge than state-cause. 

The first step in a decision analysis is the elicitation of problem elements 
and their relationships. Text analysis with decision trees as the target struc¬ 
ture has been used by Gallhofer and Sarris (1988). The coders are given a 
description of decision trees and instructions for their coding. The process 
requires the “bottom-up” coding of subtrees from fragments of text, the 
combination of subtrees, and then the development of an overview tree. It is 
strongly advocated here that the ability to separate the relational model 
from the numerical and functional levels will facilitate text analysis. 

The approach taken here, and supported by other researchers (Ashley, 
Stokes & Perng 1988; Bradshaw & Boose 1990; Bradshaw et al. 1991; 
Henrion 1989; Ramaprasad & Poon 1985), is to exploit the relational ability 
of influence diagrams and to build a qualitative model first. This can be 
expanded quantitatively later. The development of a qualitative structure is a 
labor- and knowledge-intensive task. Henrion (1989) reports that it took 
a day of structured interviewing by an experienced decision analyst to elicit a 
problem structure of thirty variables from a knowledgeable domain expert. 
One aim of problem structuring is to reduce the skill and time necessary to 
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produce a qualitative model while still giving the decision maker the under¬ 
standing of the problem necessary to implement the policy with confidence. 

TEXT ANALYSIS TO PRODUCE INFLUENCE DIAGRAMS 

The approach proposed here is similar to that proposed by Ramaprasad 
and Poon (1985) for building the influence diagrams used in systems dy¬ 
namics. These systems dynamics influence diagrams differ primarily from 
the Bayesian influence diagrams in that they include feedback; that is, they 
are cyclic graphs. There are, however, some useful similarities in their con¬ 
struction. They proposed a technique with the steps of specifying the frame¬ 
work, specifying elements, specifying relationships between elements, 
specifying the strength of relationships between elements, and mapping the 
systems dynamics influence diagram. 

The specification of the framework provides the categories of variables 
(they call them elements). This step is unnecessary when building Bayesian 
influence diagrams as the predetermined categories of decision, chance, and 
deterministic events and outcomes must be used. One other difference is the 
use of a matrix to show the links between elements. Ramaprasad and Poon 
(1985) claimed that this would appeal to more users. They were unable to 
provide a convincing argument to support this, and until there is reliable 
empirical evidence to the contrary, the graphic form of influence diagrams 
will be advocated. The others steps of their technique are used as they are. 

Text analysis with influence diagrams as a target structure is driven by the 
user and allows for the analysis of problems expressed in fragments of text 
such as- a number of paragraphs. Such fragments of text are an appropriate 
communication medium for managerial users. Text is analyzed and variables 
or more abstract descriptions of actions, events, and goals elicited. The key 
step in this activity is the ability to consider relationships in isolation. For 
each isolated variable, the question is asked, “Does this variable directly in¬ 
fluence another variable?” The influences and informational links that are 
isolated can be used to build influence diagrams. 

With iess-well-structured problems, the approach to modeling is primarily 
exploratory. Top-down progressive refinement is made possible by the ability 
to construct a hierarchy of influence diagrams (Rege & Agogino 1988). 
Within each fragment of text, the elements and their relationships can be 
determined in isolation from the rest of the model. The ability to represent 
problems at different levels of abstraction and the separation of the rela¬ 
tional and numerical representation help overcome human cognitive limita¬ 
tions in information processing. 

The interactive processing of diagrams at the numerical level will provide 
feedback necessary to focus attention on those parts of the problem about 
which there is least confidence or to which policy is most sensitive (Owen 
1984). Strategic management problems often require deep knowledge that 
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can generate a large search space. The ability to deal only with those ele¬ 
ments of the problem to which the outcome is sensitive can make the solu¬ 
tion of such problems feasible. 

The true potential of using text analysis depends on advances in natural 
language processing. There is difficulty in understanding unrestricted natu¬ 
ral language input. Decision analysis provides a fixed number of compo¬ 
nents upon which text analysis can focus and their integration into an 
influence diagram. The aim is to build a proper influence diagram or an 
influence diagram that provides an unambiguous view of the world (Shach- 
ter 1986). If user input is reasonably constrained to fit the target structure, 
the synthesis and checking of an influence diagram is an achievable task. 

The combination of top-down text analysis and real-time sensitivity anal¬ 
ysis within the decision analysis framework is a promising means of provid¬ 
ing the sort of effective support for the process of decision making that 
could be as easily and widely embraced as spreadsheet software. Even with¬ 
out the automation of text analysis, it may provide the key to a “cookbook”- 
style methodology that can be used, by analysts or in conjunction with some 
automation, to construct influence diagrams. 


A TEXT ANALYSIS PROCEDURE 

The overall process involves the successive analysis of text, numerical 
specification, and sensitivity analysis. The following steps can be used to 
develop a “first-cut” influence diagram. 

Step 1. Write a description of the problem in simple declarative sentences. 
This step overcomes the difficulty of reading sentences with complex clausal 
structures. 

Step 2. Identify and isolate decision elements. The major concepts can be 
listed and classified according to whether they are decisions (involve 
choices), events, or outcomes. Rarely are these elements explicitly, com¬ 
pletely, and uniformly referenced in natural language, and this step can re¬ 
quire considerable analytical skill. Typical text includes a mixture of 
concepts, synonyms of concepts, and values. This phase consists of the care¬ 
ful examination of noun phrases in the text. 

Step 3. Establish relationships between decision elements. The following 
are possible: 


• Interdependence between decisions. 

• Effect of one event on another. 

• Effect of a decision on a goal (outcome). 

• Effect of an event on a decision. 

• Effect of an event on a goal (outcome). 
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Different types of relationships can be used in combining decision elements. 
Paradice (1988) has described a number of relevant relationships: 

• Causal relationships , typically established by verb phrases equivalent to “results 
in,” “leads to,” and “initiates.” There are three conditions for a causal relationship: 
one event must precede a second event, there must be a relationship that can be 
formulated between two events, and there must be some justification for believing 
the relationship is causal. This relationship is the most frequently used. Henrion 
( 1989 ) points out that the concept of an influence is unclear but that cause-effect is 
a useful heuristic for establishing influence. 

• Derived or definitional relationships, where one event is determined from another 
action or event. For example, net income can be readily determined from sales and 
total costs. Such relationships are present for deterministic variables on an influ¬ 
ence diagram. 

• Bounding relationships, where one event determines the upper or lower limit of 
another. For example, the lower and upper limits for dividends are bounded by 
income. 

• Information relationships, where an event may have an impact on a decision. For 
example, the result of a test may be useful in determining whether a more costly 
action, such as to implement a reliability program, may be taken. 

Step 4. Draw the influence diagram. Examine each node and determine its 
type: decision, probabilistic event, or deterministic event. 

Step 5. Add the goal node. If a goal node does not exist, it may be neces¬ 
sary to add a goal and an arc from all events that influence the goal. A goal 
may not be explicitly defined, or it may simply be concealed as an event. 
Another common occurrence is that the decision may have multiple objec¬ 
tives, and there will be a number of goals. When this occurs, it is necessary 
to create a single outcome node. All other outcome nodes must be converted 
to events and linked to the single outcome node. 

At this point a first-cut influence diagram has been developed. The next 
phase is to examine each of the nodes, to check for clarity and to expand 
them if necessary 7 . 

A TEXT ANALYSIS EXAMPLE 

Following is an example of a problem description from which the rela¬ 
tional model shown in Figure 5.1 could be built: 

Example: Our main income is from insurance premiums. We aim to derive as much 
income as possible from selling insurance policies by designing attractive policies. 
The income from the insurance policies is used to generate further income from in¬ 
vestments in shares and property 7 . We also derive some income from the sale of these 
investments from time to time. We try to maximize retained earnings and to pay a 
satisfactory dividend to shareholders. Our most critical decisions are the design of 
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our policies as these affect the overall level of premiums we receive and the claims we 
must settle, and the design of our investment portfolio. 

This is a typical high-level description of a strategic problem. It embodies 
a description of the business, the goals of the business, and the major deci¬ 
sions. Such a description could be prepared by a high-level manager and 
comprehended by the decision analyst. To facilitate analysis, it is suggested 
that guidelines be prepared for the manager and an educational program be 
undertaken. A short demonstration will have a very positive impact on per¬ 
formance. These guidelines can be part of the entire process of developing 
an influence diagram from a problem or domain description. Typical analy¬ 
sis can be made as follows. 

Step 1. Rewrite as simple sentences: 

• Main income is derived from insurance premiums. 

• Attractive policy design leads to insurance premiums sales. 

• Income is derived from investments. 

• Income is derived from investments sale. 

• Retained earnings produce satisfaction. 

• Dividends produce satisfaction. 

Step 2. Isolate the decision elements: 

• Insurance premiums (probabilistic variable). 

• Investment income (probabilistic variable). 

• Dividends (decision). 

• Net income (deterministic variable). 

• Policy design (decision). 

• Portfolio design (decision). 

• Satisfaction (goal). 

Step 3. Establish relationships. In this case we have added the “tax rate 
net income” relationship. This influence, although not indicated in the origi¬ 
nal problem statement, may emerge from common business knowledge as¬ 
sociated with the definition of retained earnings: 

• Policy design insurance premiums investment. 

• Portfolio design investment income. 

• Investment sales net income. 

• Investment income net income. 

• Insurance premiums -► net income. 
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• (Tax rate net income). 

• Portfolio design -*■ investment income. 

• Portfolio design investment sales. 

• Retained earnings satisfaction. 

• Dividends satisfaction. 

Steps 4 and 5. Combine related concepts in an influence diagram and de¬ 
termine the goal node. This results in the diagram shown in Figure 5.1. 

QUANTIFYING THE INFLUENCE DIAGRAM 

After the first-cut influence diagram is constructed, the next step is to 
specify it numerically. Henrion (1989) has provided a procedure for this. 
The first step is to specify levels for each variable. For example, insurance 
income might be classified as {depressed, low, average, high, booming}. 
These can then be converted to numbers such as {1.0M, 1.4M, 1.75M, 
2.0M, 2.5M}. 

The second step is to quantify the influences between probabilistic events. 
The domain expert, by including a link in the text analysis procedure, has 
implied a conditioning influence and a direction of influence. The condi¬ 
tional probabilities for all events that influence a particular event may also 
initially be specified linguistically. Henrion (1989) has suggested a set of nu¬ 
meric values {0, 0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8, 0.9, 0.95, 0.98, 
0.99, 1.0}, which can be reliably assessed and are sufficient for most prob¬ 
lems. Methods such as those proposed by Hamm (1991) can be used to con¬ 
vert the linguistic expressions to numeric expressions of belief. 

The quantification procedure may lead to revision of the procedure. For 
example, the qualitative model may show conditional dependence between 
an event V and two other events. Cl and C2. On assessment of P(V j Cl, C2) 
it may be found that they are independent. One conclusion is that they are 
independent and the links can be removed and the diagram rechecked. An¬ 
other explanation may be the presence of ‘‘hidden variable,” H, not elicited 
in the qualitative modeling phase. For example, it may be that H is depen¬ 
dent on Cl and C2 and the appropriate assessments are P(H jCl, C2) and 
P(V j H). 

Once the model has been quantified, it should be validated by sensitivity 
analysis and testing using scenarios. 

PARTITIONING LARGE INFLUENCE DL4GRAMS 

It is aimed to keep influence diagrams small by restricting the number of 
nodes in a diagram to, say, 7 +/- 2, so that they can be easily solved. This 
also has the advantage that problem descriptions can be represented at one 
level. To describe a problem fully, it may be necessary to represent, at least 
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initially, a large number of nodes. Not all of these nodes need to be consid¬ 
ered at once; however, they represent a set of nodes that may be drawn in to 
the problem structure if necessary. These larger representations may be nec¬ 
essary to assure users that all major problem components have been iso¬ 
lated. 

The partitioning of problems into subproblems is a useful mechanism for 
maintaining completeness while allowing users to access and understand de¬ 
tails. Heckerman (1990) has described similarity networks for diagnosis 
problems. Similarity networks can be used to decompose large subproblems 
when the influence diagram contains only a single decision (the diagnosis) 
and a large number of chance nodes. 

A more general method is required for constructing influence diagrams 
for less-well-structured problems. A top-down approach is proposed here. A 
first-level diagram is developed to isolate major events, decision, and an out¬ 
come node. This diagram must be proper, that is, it must be a directed acy¬ 
clic graph (Shachter 1986) with a single outcome node. It is proposed that 
partitioning be based on the expansion of influences. 

The need to expand a model may result from attempts to assess nodes or 
as a result of deterministic or probabilistic analysis. Estimated (or historical) 
values and the variability of individual nodes can be assessed. The model 
may reveal that insurance premium income is the most important variable 
contribution to net income but that the effect of policy design on insurance 
income is too crude. The influence “Policy Design Decisions” “Insurance 
Income” could be expanded through text analysis to provide a subdiagram 
that includes further significant model elements, such as reinsurance effects 
claims and costs, as shown in Figure 5.2. 

The shaded nodes are those that are included in the higher-level diagram. 
There are two types of shaded nodes: sources, which have only outgoing 
influences, and sinks, which have only incoming influences. When expand¬ 
ing an influence, the sending node will be a source and the receiving node a 
sink. Unshaded nodes are new nodes, introduced first at this level. These 
nodes can be assessed numerically. Subdiagrams differ from overview dia¬ 
grams and single-level diagrams in the following ways: 

1. The subdiagrams need not have an outcome node. 

2. Any node that is a sink should be shaded, that is, it should exist in a higher-level 
diagram (this is not true of source nodes), or it is a barren node and can be safely 
removed. 

3. The integration of subdiagrams and the top-level diagram should result in a 
proper influence diagram. 

The partitioning of problems is principally aimed at providing a commun¬ 
ication device for users that allows large problems to be described com¬ 
pletely. The diagram must still be solved from a complete reconstruction of 
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Figure 5.2 

Insurance Income Subdiagram 



the constituent subdiagrams to a single-level diagram. A fruitful area of fu¬ 
ture research may be to exploit the partitioning and to look for conditions 
under which subdiagrams can be solved locally. 


CONCLUSION 

A "cookbook” text analysis procedure provides a means of capturing ini¬ 
tial problem knowledge, encouraging the decision maker to think about the 
problem in a structured way. It may also be a basis for some automation of 
problem structuring in combination with advances in natural language 
processing. Combined with the model refinement heuristics (for example, 
Moore & Agogino 1987), it may be possible to develop software that widely 
supports the process of decision making. 

Such software must be able to handle very different levels of abstraction 
from the initial sketchy description to a highly detailed and quantified anal¬ 
ysis. Strong linguistic abilities would be desirable, to build models from 
complex fragments of text by referencing factual and semantic knowledge 
from domain databases, commonsense business knowledge bases, and pre¬ 
vious decisions. 
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Computational linguistics is hard, and unrestricted natural language re¬ 
quires semantic analysis beyond the current capabilities of computer sci¬ 
ence. Even without automation, text analysis can be the basis of a manual 
method for the development of decision models much in the way that data 
models are constructed. Decision makers will gain the benefit of problem 
understanding, as well as that of improved decisions. 
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Part Three 


Spreadsheet-based Software 

The essence of decision-aiding software that is based on spreadsheets 
and multicriteria decision making (MCDM) is to have (1) the goals to be 
achieved on the columns, (2) the alternatives available for achieving 
them on the rows, (3) the relations between goals and alternatives in the 
cells, (4) an overall total for each alternative at the far right, and (5) the 
capability for quickly determining the effects of changes in the inputs on 
those overall totals. 

That type of decision-aiding software can come in various forms. One 
way of distinguishing between the forms is in how they handle the basic 
criteria for comparing software packages mentioned in the Introduction 
to Part One of this book. Other less important ways of classifying 
spreadsheet-based software might be in terms of the brand name of the 
spreadsheets used, such as Lotus 1-2-3, Quattro, SuperCalc, VisiCalc, 
Framework, and Excel. 

On the matter of multiple dimensions on multiple goals, a frequent 
approach is to measure all the goals on a common scale —for example, 
the 1, 3, 5, 7, and 9 scale used by Thomas Saaty in his analytic hierarchy 
system, the 0-100 scale used in Light Year and some other commercial 
packages, or the 1-5 scale used in P/G% drawing upon the scale theory 
developed by Rensis Likert. Those approaches, however, waste useful in¬ 
formation if measurement is available indicating how many dollars each 
alternative costs or how many hours each alternative takes. The P/G% 
system is one of the few that allows diverse measures to be preserved by 
handling the multidimensionality through raw score increments, raw 
score weights, and part/whole percentages. The percentaging approach 
converts the miles, hours, dollars, and other measures into percentages 
of the maximum obtainable, the total obtainable, or some other per¬ 
centage base that provides for a norming across diverse goals. 

On the matter of missing information, a frequent approach is to seek 
the missing information, although doing so may be unnecessarily expen¬ 
sive. An alternative is to insert an average score, which may be unneces¬ 
sarily invalid. An increasingly frequent approach is to use a variation on 
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whai-if analysis, whereby the user experiments with different relation 
scores, goal weights, or other missing items to see the effects if the un¬ 
known variable were scored in various ways. That sounds sophisticated 
and is a big selling point of Lotus 1-2-3 and other commercial spread¬ 
sheets. It is better, however, to try to arrive at a set of break-even, thresh¬ 
old, critical, or other tie-causing scores above which one alternative is 
the winning one an d below which another alternative would win. If, for 
example, the threshold value of a missing item is 6, then the user is in¬ 
formed that any number above 6 will enable the tentative first-place win¬ 
ner to retain first place, and any number below 6 will convert the 
second-place alternative into the new first-place winner. That kind of 
information can be provided for all the inputs. One then no longer asks 
what is the true value of a missing item. Instead, one asks the easier 
question of whether the true value is likely to be above or below the 
threshold value of 6 (or whatever else the threshold value might be). Var¬ 
iations on that kind of threshold analysis can apply to two, three, or 
more missing items simultaneously. 

On the matter of allocation of money, time, people, or other re¬ 
sources to places, activities, or other objects, the traditional approach 
tends to be a highly subjective holistic perspective in which the goals to 
be achieved are only implicit. At the other extreme is the perspective of 
linear, nonlinear, or other forms of mathematical programming in 
which the only goals are those that are highly measurable so that equa¬ 
tions and inequalities can be worked with. Spreadsheet software with an 
MCDM base (rather than a linear programming base) can enable a user 
who is not mathematically oriented to be able to deal with goals and 
relation scores, while at the same time the system can implicitly satisfy 
the rules of classical calculus optimization. 

This is true of the pan/whole percentaging system where one allo¬ 
cates to each budget category in proportion to how well it does on a 
weighted sum of their relations with explicit goals, where the relation 
scores can be on a scale as simple as a 1-5 scale. The system uses the 
ratios between such relation scores for different budget categories as 
proxies for the ratios between nonlinear elasticity coefficients. Such co¬ 
efficients are virtually impossible to obtain directly given the nonavaila¬ 
bility of data and the inability to control for confounding factors. The 
system is capable of dealing with diminishing returns whereby the best 
alternative receives the highest allocation but not 100 percent of the re¬ 
sources. At the same time, this nonlinear system does not get stuck in 
nonoptimum ruts since the system provides analytic solutions rather 
than wild reiterative guessing. 

On the matter of satisfying multiple constraints, especially conflict¬ 
ing constraints, the traditional approach is to provide for some kind of 
compromise whereby each constraint may be partly violated or less im¬ 
portant constraints might be completely violated. At the opposite 
extreme is the management science/operations research (MS/OR) ap¬ 
proach of simply saying there is no solution if there are conflicting con¬ 
straints. Spreadsheet softw'are with an MCDM base helps facilitate the 
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finding of solutions where all the conflicting constraints can be simulta¬ 
neously satisfied. One way it can do so is by clarifying exactly how much 
the budget needs to increase so that all the constraints will be satisfied. 
Another way is by clarifying how much the efficiency of each alternative 
has to improve on each goal in order to satisfy all the constraints with¬ 
out increasing the budget. A third way is by showing the combinations 
of budget increases and efficiency improvements that can simultane¬ 
ously satisfy all the constraints. The MCDM spreadsheet approach can 
also clarify how new alternatives or new combinations can provide su¬ 
peroptimum solutions whereby all major constraints or viewpoints ex¬ 
ceed their best initial expectations. The software does that by facilitating 
trial and error, by clarifying the characteristics of superoptimum solu¬ 
tions, and by enabling one to keep better track of the optimizing analy¬ 
sis when there may be many goals and many alternatives present. 

On the matter of making predictions of the effects of alternatives on 
goals, the traditional perspective tends to be rather subjective and specu¬ 
lative. The MS/OR perspective looks technically competent with its use 
of multiple regression analysis, sophisticated time series, and other 
quantitative forecasting techniques. Those techniques, however, tend to 
ignore important predictive criteria that are not highly measurable. They 
also tend to be at the mercy of reciprocal causation, spurious causation, 
and interaction factors. The scores one inserts into the cells of an 
MCDM spreadsheet tend to rely more on common knowledge and de¬ 
duced relations from known facts or reasonable premises. As with pre¬ 
scriptive analysis, the predictive analysis of MCDM spreadsheet 
software facilitates trial and error in order to arrive at a more valid and 
simpler predictive decision-rule by experimenting with different predic¬ 
tive criteria, weights, measurement units, and other inputs. 

On the matter of simplicity, traditional lay approaches to decision 
making may be overly simple. Often they involve only one alternative, as 
in a go/no-go situation, and only one criterion, business profit. The de¬ 
cision then reduces to whether there is a positive or negative relation 
between the alternative and the criterion, frequently ignoring other al¬ 
ternatives, other criteria, constraints, and other methods of analysis be¬ 
sides dichotomous choice. At the other extreme is the overly complex 
MS/OR perspective, which may also have only a single objective func¬ 
tion and a go/no-go alternative, but they are processed by way of elabo¬ 
rate models from mathematical programming, four-cell payoff matrices, 
decision trees, and network diagrams. The MCDM spreadsheet com¬ 
bines the validity of recognizing multiple alternatives and criteria with a 
simplicity that lay users can understand. A decision analysis table can be 
grasped by people with reasonable common sense and no technical 
background in mathematics, statistics, operations research, manage¬ 
ment science, or related fields. 

Although the MCDM-based spreadsheet approach may have many 
advantages on an abstract level, there may be many situations where tra¬ 
ditional lay approaches or complicated MS/OR approaches are more 
applicable. The traditional lay approach of the chicken crossing the road 
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usually works fine. One would not expect either chickens or people to do 
an MCDM spreadsheet analysis to decide whether or when to cross the 
street. That problem has been handled reasonably well ever since there 
were streets. It can be improved upon by looking both ways but not nec¬ 
essarily by making use of decision-aiding software. The problem is too 
simple, too frequent, too well done already, and not much is usually at 
stake. If one makes the mistake of not crossing when one could have 
gotten across, one can still cross two minutes later. There is not only one 
chance. If one makes the mistake of crossing when it is premature to do 
so, one can generally scurry back to the curb. A majority of all decisions 
are made by such holistic gut reactions. 

Similarly, If an engineer is faced with a problem of how to build a 
bridge that will satisfy a lot of goals like safety, inexpensiveness, and 
wideness for carrying lots of cars, then a mathematical approach may be 
more relevant than a spreadsheet matrix. The mathematical approach 
may involve a form of linear or nonlinear programming that will take 
into consideration budget constraints, safety constraints, and car speed 
constraints, and not just goals of saving money, lives, and time. MS/OR 
models work especially well when many different kinds of constraints 
are present that can be quantified in the form of a series of inequalities. 
Trying to work with such constraints using MCDM spreadsheet perspec¬ 
tives involves ioo much trial and error, which may result in causing one 
to think one has arrived at an optimum solution only because one has 
put in a lot of time trying out nonoptimum alternatives. 

The key problem in this contest is that people in business may tend 
wrongly to emulate the people in engineering in seeking to find profit- 
maximizing equations. Worse than that, people in public policy may 
wrongly seek to emulate both the engineers and the quantitative busi¬ 
ness economists in trying to find equations for dealing with problems of 
free speech, equal treatment under law, fair procedure in criminal jus¬ 
tice, and other such public policy problems. In those areas the MCDM 
spreadsheet perspective is at its best given its ability to deal with multiple 
dimensions, missing information, resource allocation, conflicting con¬ 
straints, subjective prediction, and the need for simplicity in drawing 
and presenting conclusions. 
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Using computer technology to enhance the decision maker’s ability to iden¬ 
tify, evaluate, and implement decisions has long been a goal of computer 
scientists and decision makers. Through the 1950s, 1960s, and 1970s, tre¬ 
mendous strides were made in the business use of computers as computer 
capabilities expanded and costs declined. During this period, it became 
commonplace for large corporations to own powerful mainframe computers 
and to devote extensive human resources to the support of those tools. 

Despite these advances, only small steps were made toward achieving the 
visionary goal of unlocking individual decision maker creativity through 
computer use. Because of the high cost and specialized skills needed to oper¬ 
ate mainframe computers, the use of these tools was dominated by data 
processing and information management activities. Although using com¬ 
puters to manage payrolls, monitor inventory levels, and prepare routine fi¬ 
nancial statements can improve business efficiency, the gains tend to arise 
from cost reductions rather than advances in manager productivity. Further, 
the high acquisition and support costs of the mainframe computer preclude 
their extensive use in small and medium-sized firms, as well as by indi¬ 
viduals. 

During the latter portion of the 1970s, the personal computer revolution 
burst into being. Suddenly the financial barriers separating the individual 
decision maker and computer capabilities were effectively destroyed. For 
much less than the price of a personal automobile, individuals could acquire 
microcomputers with significant computational capacity. And as personal 
computers advanced in capacity and declined in cost in the 1980s, even more 
effective tools were made available. 

But significant barriers still remained, impeding creative decision-making 
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applications of the computer, especially by individuals and managers who 
were either too busy or not inclined to experiment with computers. Effective 
software was soon recognized as a major barrier. Potential users were faced 
with the choice of purchasing off-the-shelf packages or developing their 
own applications software. Adequate software packages for data process¬ 
ing-oriented activities, especially general ledger accounting, soon became 
available. Data processing with the microcomputer, however, delivers little 
more decision creativity than do the same activities performed with large 
computers. And although higher-level programming languages, such as 
BASIC or Pascal are easier to learn than earlier mainframe languages, their 
effective use requires extensive learning time, and their application is seldom 
intuitive to the casual user. 

In the entrepreneurial world of the personal computer, such an unmet 
need did not exist long. General-purpose software packages soon became 
available. General-purpose spreadsheet and database packages promised 
that the nonprogrammer could use the personal computer to develop signifi¬ 
cant decision support system (DSS) capabilities. These DSSs could be suffi¬ 
ciently sophisticated to address important business problems. Further, the 
ease of use of general-purpose software implied that tailored DSSs could be 
developed. Tailoring of the DSS to accommodate the unique features of spe¬ 
cific decision situations as well as the desires and needs of individual deci¬ 
sion makers is critically Important in moving to an environment where DSSs 
truly support creative decision processes. 

Although these technological advances have presented individual decision 
makers with an unprecedented opportunity, much remains to be learned to 
employ most effectively these powerful tools to aid decision making. This 
chapter draws upon a very unusual educational experience to gain insights as 
to both the potential for DSS development offered by these tools and the 
significant Impediments that still exist. Specifically it addresses the follow¬ 
ing questions: 

1. Can useful DSSs, applied to nontrivial business problems, be developed by non¬ 
programmers using a general-purpose software package such as Lotus 1-2-3? 

2. Can such spreadsheet-based tools be developed so as to overcome psychologically 
based limitations of human decision making, such as those arising from memory 
capacity constraints or from our weaknesses in addressing the uncertain nature of 
business decision situations? 


EDUCATING STUDENTS ABOUT 
SPREADSHEET-BASED DSSs 

Because of burgeoning student interest in microcomputers and to investi¬ 
gate the potential for DSS development, an experimental class was started at 
the University of Illinois at Urbana-Champaign in the spring of 1986. The 
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class, titled Decision Support Systems in Agriculture, took an aggressive 
posture as to the most effective means to advance student understanding of 
DSSs and the microcomputer technology. Instead of just reviewing systems 
available, each student was charged with developing a spreadsheet-based 
DSS that addressed a realistic business decision problem and incorporated 
relevant decision concepts from the rapidly advancing field of decision the¬ 
ory. This philosophy resulted in an educational experience in the use and 
application of microcomputers. 

In the four times that the course has been offered, over eighty students 
have participated in the class. These students were upperclass undergradu¬ 
ates or master’s students interested in management of agricultural firms. Al¬ 
though a few plan to be directly involved in farm production, the majority 
will pursue careers in the large and diverse food and agribusiness sector. Stu¬ 
dents in the class have taken positions as loan officers, in sales and market¬ 
ing, and as merchandisers involved with commodity trading. Direct 
development of computer software has rarely been a career goal of a stu¬ 
dent. 

Introductory computer competence and familiarity with spreadsheets is a 
prerequisite for entry into the course. Prior to the class, the typical student is 
comfortable entering data into spreadsheet programs, constructing simple 
formulas, and performing routine file maintenance and printing activities 
with a spreadsheet. If any prior coursework in programming had been 
taken, it would have been limited to an introductory course in programming 
using a language such as Pascal or BASIC. 

The course structure is divided equally between gaining the capability to 
employ advanced spreadsheet features (macros, menus, and database com¬ 
mands) and developing an understanding of relevant problem solving and 
decision theory concepts. For the latter goal a diverse set of readings from 
business and psychology literatures is reviewed and discussed. As a term 
project, each student develops a prototype DSS that addresses a realistic 
business situation and illustrates relevant aspects of decision making within 
the framework of the DSS. 

PROBLEM SOLVING AND DECISION THEORY CONCEPTS 

Identification of realistic decision situations can often be a problem for 
college students; however, because all of the students are focusing on agri¬ 
cultural careers, their access to decision makers and their familiarity with 
typical agribusiness problems is improved. In addition, many of the students 
are employed part time or have an involvement with a family-operated busi¬ 
ness. In fact, nearly 50 percent of the projects developed have been created 
for a specific decision maker rather than for a hypothetical situation. Exam¬ 
ination of the types of problems addressed may give an indication of the 
likely uses of spreadsheet-based DSSs in the business world. 


130 


Computer-Aided Decision Analysis 


Types of Problems 

Problem settings can be categorized as strategic, tactical, or operational in 
nature. The spreadsheet DSSs developed have had a significant representa¬ 
tion from all three categories. DSSs devoted to strategic choices have consid¬ 
ered the desirability of major capital asset purchases, start-up of new 
business ventures, and investigation of the performance of specific subunits 
within a business. Tactical decisions evaluated have included income tax 
planning, choice of financing options, and government farm program par¬ 
ticipation. DSSs addressing tactical and strategic problems were designed 
for relatively unstructured decision situations. A wide range of operational 
decisions has been evaluated, including comparison of production alterna¬ 
tives, grain marketing options, and alternative purchasing sources for in¬ 
puts. These operational DSSs are typically employed in structured decision 
settings. 

The majority of the DSSs have included a financial analysis among alter¬ 
natives. These utilize projection methods to perform what-if evaluations. A 
number of these programs have included a significant data query capability 
as well. For example, a DSS designed for an agricultural consulting firm 
assists in monitoring employee hours but also provides the capability to per¬ 
form numerous sorting options. With this capability, the project manager 
can compare planned to actual performance by project, by client, or by type 
of activity. 


Decision Theory Concepts 

Over the past few decades, there has been a considerable advance in our 
understanding of decision-making processes of individuals. Although a sub¬ 
ject of debate between some economists and psychologists, it is generally 
conceded that the pure assumptions of the completely rational economic 
person are implausible (Simon 1987; Arrow 1982). Rather, the search for 
information is costly, decision-maker time is limited, and their attention of¬ 
ten is fragmented. DSSs exist to improve the decision making of individual 
decision makers. To do so requires that such programs explicitly consider 
human limitations and constraints to decision making. 

One of the goals of the course described was to determine if decision the¬ 
ory concepts could be incorporated into spreadsheet-based DSSs. Three 
types of concepts have been considered: the first deals with the physical op¬ 
eration and use of spreadsheets, the second arises from the limited memory 
capacity of humans, and the third relates to the difficulty that decision 
makers have in assessing situations in which significant uncertainty exists. 

When introduced, spreadsheets immediately were welcomed because of 
their intuitive nature (Sonka 1983). Menus were available to assist the user 
by summarizing a series of commands into a few keystrokes, and their use of 
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words as prompts allowed the infrequent user to recall commands rather 
than having to memorize them. However, as spreadsheets were applied to 
increasingly sophisticated problems, a number of difficulties emerged. Cur¬ 
sor movement becomes difficult and tedious as the spreadsheet grows larger. 
Further, the letter/number combination of cell coordinates is not particu¬ 
larly intuitive, nor is it easy to remember. 

To overcome these difficulties, spreadsheet-based DSSs can be designed to 
be user friendly through extensive menus. These menus limit the need to 
scroll across the spreadsheet with the cursor and allow the user to perform 
complex operations (such as printing a set of reports) with minimal use of 
keystrokes. 

Although often thought of as the ultimate computer, the human mind has 
limits in the amount of information it can process and remember. Effective 
DSS design should assist the user in overcoming these limitations. Davis and 
Olson (1985) catalog a substantial list of such limitations, among them, in¬ 
formation overload, the need for feedback, filtering, and the process of an¬ 
choring and adjustment. Information overload, as its name implies, refers 
to the decision maker’s ability to comprehend and process an excessive 
amount of information. Report format is an important aspect of overload. 
For example, a manager accustomed to evaluating a specific form of finan¬ 
cial statement may find information presented in that form easy to assimi¬ 
late, although the report may appear cluttered to others. 

To feel comfortable with the technology, users need feedback as they use 
computers. In DSS design, construction of compact input forms so that the 
user attains a feeling of accomplishment by completing separate sections is 
one means of providing feedback. At times, spreadsheets need to perform a 
number of calculations or manipulations, and in doing so the user may per¬ 
ceive a loss of control. Spreadsheet-based DSSs should anticipate these situ¬ 
ations and provide messages on the screen to reassure users that normal 
processes are occurring. 

Filtering is a process by which large amounts of data are reduced to a 
more useful form. The definition of useful is critical in this summarization. 
Therefore, DSS developers need to understand the information needs of the 
user. This aspect is one of the major potential benefits that spreadsheet- 
based DSSs offer because in many cases the DSS developer can also be a 
user of the information. 

Anchoring and adjustment refers to the process by which decision makers 
evaluate information. An estimate of next year’s net income, for example, is 
of little value unless the user can place that number into a business context. 
Decision makers automatically attempt to make a comparison between that 
estimated income and last year’s income. In this case, last year’s income is 
the anchor point, and the manager considers the estimated income as an 
adjustment from that anchor. DSSs may significantly assist decision making 
if they can provide appropriate and accurate anchor points for the user. 
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Most significant business decisions involve uncertain outcomes, and man¬ 
agers must make decisions without a guaranteed outcome. Unfortunately, it 
has been widely documented that individuals are not, in a practical sense, 
good statisticians and tend not to follow rules of logical decision making 
when outcomes are uncertain. Kahneman and Tversky (1979) have shown 
that problem framing is a critical part of the decision-making process. Their 
prospect theory approach suggests that design of DSSs must pay particular 
attention to whether the manner in which the problem is framed will affect 
the user’s response to its information. 

Chance events can dramatically affect outcomes of nearly every project. 
Yet decision makers have difficulty in separately identifying the role of 
chance from that of performance in evaluating outcomes. DSSs are widely 
recognized for having the ability to demonstrate the effect of chance events 
on projected outcomes. Further, DSSs should also be effective in helping to 
overcome the hindsight bias that can distort evaluation of outcomes. 

EXAMPLE DSSs 

A large number of prototype DSSs have been developed in the Decision 
Support Systems in Agriculture class. Here three specific spreadsheet-based 
DSSs will be briefly reviewed. These three were selected to illustrate spread¬ 
sheet-based DSS application across a range of decision situations. The first 
of the three will be discussed in greater detail, to illustrate the potential for 
user-created menus to assist the decision maker. Discussion of the two re¬ 
maining DSSs will be limited to an examination of their purposes and the 
means by which they relate to some of the decision concepts noted above. 

CAP P's DSS 

A manager of a farmer’s cooperative was considering alternative means to 
expand the profitability of the enterprise. The firm comprises several sub¬ 
units, of which retail and wholesale sales of fertilizer are major factors. Ex¬ 
panding the retail sales component seemed a likely candidate for success, 
but numerous uncertainties existed. A major limitation was that the firm did 
not have adequately detailed cost records to assess accurately the indepen¬ 
dent profitability of the retail fertilizer operation. The CAPP’s RFD (cost 
allocation and projected profitability of the retail fertilizer division) DSS 
was developed to assist that manager in this assessment of strategic options. 
In particular the manager wanted a tool to examine some of the implications 
of expanding retail fertilizer sales prior to conducting an extensive and ex¬ 
pensive analysis of that option. 

CAPP’s RFD allows the manager to utilize existing data, make minimal 
assumptions, and observe the impact of those assumptions on the expected 
profitability of fertilizer sales. Because the manager involved had minimal 



Spreadsheet-based Decision Support 


133 


computer background, the DSS needed to be extremely easy to use. This 
goal was accomplished through extensive use of specialized menus created 
using the Lotus 1-2-3 macro and command language capability. 

Figure 6.1 presents the main menu that the user sees when initiating the 
CAPP’s RFD program. The words listed as “menu options to choose 
among” in the figure are the options that the user sees in the panel area of 
the Lotus 1-2-3 screen. These options operate in an identical fashion to Lo¬ 
tus l-2-3 5 s general menu options; the user can invoke an operation by mov¬ 
ing the cursor to be positioned over an option and hitting the enter key or by 
typing the first letter of that option (e.g., p for print). To allow the user to 
understand better the choices available, the messages listed under “explana¬ 
tion of menu options” are shown in the window area of the screen. (The 
window area is the main part of the screen when one views a Lotus 1-2-3 
spreadsheet.) In addition to providing additional information, presenting 
the menu option explanations in the window area ensures that the user is not 
distracted or confused by extraneous information that may appear on the 
screen. 

After selecting a menu option, the user may immediately proceed to an 
active worksheet area (moving directly to a help screen with additional infor¬ 
mation) or is presented with another set of options within a submenu. Fig¬ 
ure 6.2 presents a schematic of the significant submenus incorporated within 
CAPP’s RFD. (Opportunities to back out of the save and quit options are 
available but are not shown here.) After selecting the begin option, for ex¬ 
ample, the appropriate submenu appears. Here the user can choose to input 

Figure 6.1 

Example of Main Menu Screen of the CAPP’s RFD Spreadsheet DSS 

Menu options to choose among 

BEGIN LOOK PRINT SAVE QUIT DOC HELP 


Explanation of menu options 

BEGIN using CAPPs RFD: Input data. 

LOOK at results on screen. 

PRINT results. 

SAVE worksheet. 

QUIT using CAPPs RFD. 

Documentation. 

HELP directions for using CAPPs RFD. 
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Figure 6.2 

Complete Menu Structure for the CAPP’s RFD Spreadsheet DSS 

MAIN MENU 

_.BEGIN LOOK PRINT SAVE QUIT DOC HELP 

\ T PRINT SUBMENU 

P & L COSTS MARGINS INPUTS RETURN DOC 


P & L COSTS MARGINS RETURN 


MARGINS EXPENSES GROWTH RETURN HELP 


i^.EXPENSES SUBMENU 

GENERAL LABOR PLANT TAXES ADMIN I NT & DEP RETURN 


[ MARGINS SUBMENU 

GRAIN FERTILIZER TRENDS ADDITIONAL RETURN 

data associated with margins on the firm’s operations, expenses, projections 
for growth, return to the main menu, or examine a help screen. If either the 
margins or expenses options are chosen, submenus with a number of addi¬ 
tional choices appear. 

With the menu structure shown in Figure 6.2, CAPP’s RFD is a com¬ 
pletely menu-driven program. Figure 6.2 illustrates the level of complexity 
for the DSSs developed in the Decision Support Systems in Agriculture 
class. This level also is representative of that needed to attack meaningful 
decision problems with spreadsheet-based DSSs. 

CAPP’s RFD prepares a four-year projection of financial performance. 
Table 6.1 shows the key base data needed on margins for the firm. Here the 
base year is shown as 1987. (The data shown are completely fictitious to 
preserve the financial integrity of the firm involved in this DSS develop¬ 
ment.) To enter or alter a number, the user moves the cursor to the appropri¬ 
ate cell and types in the desired value. To aid the user, cursor movement is 
restricted to cells in which values should be entered. In CAPP’s RFD, for 
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Table 6.1 

Screen to Input Base Year Margins 

RETAIL FERTILIZER REVENUE 


Input MARGINS and TONNAGE incurred in 
for each”item. Press ENTER when finished. 


1987 


Fertilizer Margin per Ton 

Dry Fertilizer $4.50 

Liquid Fertilizer 6.00 

Anhydrous 40.00 

Limestone 3.00 

Spreading & Delivery 2,500.00 
Misc. (total) $10,000.00 


Tonnage Sold 
10,000.00 tons 
1,500.00 tons 
10.00 tons 
600.00 tons 


example, the cursor can only move to the six cells under the “Margin” 
column and the four cells under the “Tonnage Sold” column. 

A major problem limiting the firm’s strategy evaluation was lack of de¬ 
tailed cost allocation data. Table 6.2 presents the table in which general allo¬ 
cation proportions are entered. This table appears after the user selects the 
GENERAL option under the EXPENSES submenu (Figure 6.2). Although 
the user does not have precise allocation data, use of the proportions ap¬ 
proach in Table 6.2 allows the user to conduct preliminary evaluations with- 


Table 6.2 

Screen to Input General Allocation Factors 

GENERAL ALLOCATION SCHEDULE 

Enter the PROPORTION of each expense category you wish to have 
allocated to the retail fertilizer division. You will have a chance 
later to make specific allocations if you wish. ENTER to return. 


Category Proportion 

Management & Labor 0.70 

Plant Expenses 0.60 

Taxes, Insurance, Leases 0.65 

General Administrative 0.75 


Interest & Depreciation 


0.50 
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out undue expense to acquire more precise data. These proportions are 
easily changed so that the manager can observe where data shortcomings are 
most significant. (Again, the cursor is constrained to moving within the five 
cells under the proportion column.) 

Table 6.3 illustrates the format of the tables within which specific expense 
data are entered. (This table is reached by selecting the LABOR option 
within the EXPENSES submenu in Figure 6.2). Expenses for both the retail 
and wholesale fertilizer divisions are fairly well known by the firm’s man¬ 
ager, and these values are to be entered under the “Total Expense” column. 
The data in the “Proportions” column are automatically transferred from 
the general allocation schedule (Table 6.2) unless the user overrides that 
allocation for a specific expense category. To illustrate, the 0.50 proportions 
for the managers 5 salaries and office salaries categories have been manually 
entered, whereas the 0.70 proportions for the other categories were trans¬ 
ferred from the management and labor category of Table 6.2. 

The major output report of CAPP’s RFD is a five-year profit-and-loss 
statement (Table 6.4). The first data column presents actual profit data for 
1987. This serves as an anchor point by which the user can evaluate the im¬ 
plications of the four-year projection estimates shown in the remaining 
columns. The volume data at the bottom of the figure serve as an alternative 
reference point, which may be more meaningful for board members who 
may not be comfortable with financial data only. The profit-and-loss state¬ 
ment shown in Table 6.4 is somewhat crowded and suffers to some extent 

Table 6.3 

Screen to Input Specific Cost Data 

MANAGEMENT AND LABOR COSTS 

Enter TOTAL EXPENSE expected or incurred and the PROPORTION to 
allocated to retail fertilizer in 1987; ENTER when finished. 


Managers Salaries 
Office Salaries 
Plant Salaries 
Payroll Taxes 
Employees' Benefits 
Employee Education 
Sales Commissions 
Pension Fund Account 
Miscellaneous 


Total Proportion 


Expense 


$30,000.00 

0.50 

2 # 000.00 

0.50 

15,000.00 

0.70 

1,500.00 

0.70 

1,000.00 

0.70 

200.00 

0.70 

0.00 

0.70 

1,500.00 

0.70 

0.00 

0.70 
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Table 6.4 

Example of Major Output Report 


Profit and Loss Statement 



1987 

1988 

1989 

1990 

1991 

Margins 

Dry Fertilizer ....... 

$45,000.00 

$47,891.25 

$50,720.18 

$53,487.73 

$56,194.87 

Liquid Fertilizer ... 

9,000.00 

12,120.00 

15,301.50 

18,545.42 

21,852.68 

Anhydrous . 

400.00 

4,400.00 

8,400.00 

12,400.00 

16,400.00 

Licestone . 

1,800.00 

3,267.00 

4,704.48 

6,112.88 

7,492.65 

Service & Misc. 

12,500.00 

12,700.00 

13,050.00 

13,450.00 

14,000.00 

Total Gross Margin .... 

$68,700.00 

$80,378.25 

$92,176.16 

$103,996.03 

$115,940.20 

Expenses 

Management & Labor .. 

$35,840.00 

$36,656.80 

$37,739.94 

$38,744.73 

$39,619.63 

Plant Expenses . 

15,300.00 

15,606.00 

15,918.12 

16,236.48 

16,561.21 

Taxes, Insurance & 

Leases . 

1,820.00 

1,856.40 

1,893.53 

1,931.40 

1,970.03 

Gen. Administrative . 

2,572.50 

2,623.95 

2,676.43 

2,729.96 

2,784.56 

Depreciation . 

5,000.00 

5,100.00 

5,202.00 

5,306.04 

5,412.16 

Interest . 

6,250.00 

6,375.00 

6,502.50 

6,632.55 

6,765.20 

Total Retail Expense .. 

$66,782.50 

$68,218.15 

$69,932.51 

$71,581.16 

$73,112.79 

NET INCOME . 

$ 1,917.50 

$12,160.10 

$22,243.64 

$32,414.87 

$42,827.41 

Total Volumes (tons) 

1987 

1988 

1989 

1990 

1991 

Dry Fertilizer 

. 10,000 

10,750 

11,500 

12,250 

13,000 

Liquid Fertilizer . .. . 

. 1,500 

2,000 

2,500 

3,000 

3,500 

Anhydrous .. . . . . 

10 

110 

210 

310 

410 

Limestone . 

600 

1,100 

1,600 

2,100 

2,600 

Total Tonnage . 

. 12,110 

13,960 

15,810 

17,660 

19,510 


from information overload. This format is, however, the standard for the 
firm, and, presumably, managers are accustomed to evaluating information 
in this form. 

Even if complete historic cost data were available, any four-year projec¬ 
tion involves considerable uncertainty. To provide the decision maker a 
means to evaluate the effect of alternative assumptions, a number of sensi¬ 
tivity analysis tables are provided. Table 6.5 displays one such table, illus¬ 
trating the impact on expected net income of alternative margins for dry 
fertilizer products. The base margin of $4.50 was entered by the user in the 
margins table (Table 6.1). If a differing base margin is entered, the margin 
values in the sensitivity table are altered and the table recalculated. 

CAPP’s RED is an example of a spreadsheet-based DSS that can be ap¬ 
plied to the very unstructured problem of determining a firm’s future strat¬ 
egy. Its menu-driven structure, as well as that of the two DSSs yet to be 
discussed, allows the user to interface with the DSS and overcome some of 
the user-unfriendly features of spreadsheets. The program’s structure allows 
the user to consider alternative cost allocation options rapidly and in so do- 
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Table 6.5 

Example of Sensitivity Analysis Capabilities within CAPP s RFD 

SENSITIVITY ANALYSIS OF KARGINS 
FOR DRY FERTILIZER 


This table shows expected net incomes from varying margin levels, 
in each of the respective years. ENTER when finished. 


Percent of 

current 

Kargin 

1987 

1988 

1989 

1990 

1991 

85% 

$3.82 

( 4 , 832 ) 

4,976 

14,635 

24,391 

34,398 

90 

4.05 

( 2 , 582 ) 

7,370 

17,171 

27,066 

37,207 

95 

4.27 

( 332 ) 

9,765 

19,707 

29,740 

40,017 

ivu 

4.50 

1,917 

12,160 

22,243 

32,414 

42,827 

105 

4.73 

4,167 

14,554 

24,779 

35,089 

45,637 

110 

4.95 

6,417 

16,949 

27,315 

37,763 

48,446 


ing provides an effective tool to aid the decision maker in framing the prob¬ 
lem situation. Considerable filtering occurs in collapsing numerous types of 
input into a few focused output reports. 

MKTPLANNER DSS 

The typical cash grain producer physically harvests the crop during a one- 
or two-month period. That commodity can then be stored for several 
months, allowing the producer to select the most appropriate times to de¬ 
liver the crop to market. Development of the marketing plan, therefore, is 
one of the major decisions faced each year. In making this decision, the pro¬ 
ducer must consider potential price moves (which can be quite substantial), 
as well as the need for cash to operate the business until another crop is 
harvested. The MKTPLANNER DSS was designed for a moderate-sized 
cash grain operation in the Midwest. 

The major output report of this DSS is a monthly cash flow statement. 
This financial document is widely used by producers and is of special inter¬ 
est to agricultural lenders as they evaluate the creditworthiness of potential 
borrowers. The typical farm financial statement is composed of three seg¬ 
ments: 

1. A listing of receipts expected from the firm's operations. 

2. A listing of expenditures for the coming twelve months. 

3. A financing section where cash shortages from operations can be offset by bor¬ 
rowing and cash surpluses used to repay obligations. 
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The typical monthly cash flow statement epitomizes the problems of infor¬ 
mation overload. For DSS developers, a more serious problem is that the 
cash flow statement does not integrate well with the decisions that the man¬ 
ager makes. Decision makers often are more comfortable thinking in terms 
of physical units and prices (or margins) than they are in terms of aggregate 
cash data. Therefore, the cash flow statement may not be very meaningful 
for the decision maker. This problem is not limited to agricultural producers 
and is especially relevant for managers of small businesses. 

The MKTPLANNER DSS requires standard business information as in¬ 
put. Compact, user-friendly input screens, such as shown for CAPP’s RFD, 
are used to acquire this information. The major item separating the MKT¬ 
PLANNER DSS from a typical cash flow program is the student developer’s 
inclusion of a decision-making table. Thble 6.6 illustrates the format of the 
table for the first five months of the year. 

The manager for whom MKTPLANNER was created was primarily inter¬ 
ested in a tool to consider sales options in the context of financing needs for 
each option. Therefore the decision-making table allows the user to alter the 
planned timing and amount of grain sales and observe the effect on the 
firm’s monthly cash position. Further, financing alternatives, such as bor- 


Table 6.6 

Example of the Decision-Making Table of MKTPLANNER 



MON 

T H L 

Y D E 

C I S I 

O N S 


JAN 

FEB 

MARCH 

APR 

MAY 

Begining Cash Balance 

500 

4,360 

1,220 

480 

701 

Corn in Storage 

16,000 

16,000 

4,500 

3,500 

1,000 

Corn in Storage to Sell 

4,000 

0 

11,500 

1,000 

2,500 

Soybeans in Storage 

0 

0 

0 

0 

0 

Soybeans in Storage to Sell 
Wheat in Storage 

Wheat in Storage to Sell 

0 

0 

0 

0 

0 

TOTAL RECEIPTS: 

8,500 

4,360 

24,220 

2,480 

5,701 

TOTAL EXPENDITURES: 

3,765 

2,765 

8,365 

18,529 

4,429 

Operating Loan Balance 

45,000 

45,000 

30,000 

30,000 

30,000 

Repay Operating Loan 

0 


15,000 



Savings Balance 

217 

235 

254 

275 

298 

+/- from Savings 

0 

0 

0 

0 

0 

Operating Loan 

0 

0 

0 

0 

0 

Long Term Loan 

0 

0 

0 

0 

0 

ENDING BALANCE 

4,360 

1,220 

480 

701 

1,022 




140 


Computer-Aided Decision Analysis 


rowing additional operating funds or debt repayment, are also entered 
within this table. 

In Table 6.6, the rows in italics are areas where the user can enter market¬ 
ing or finance-related decisions. The upper six shaded rows refer to market¬ 
ing choices. The user is informed as to the amount of each crop in storage at 
the start of each month and then indicates the amount of desired sales dur¬ 
ing the month. The DSS automatically computes the revenue from those 
sales in determining the firm’s planned net cash position and calculates the 
amount remaining for sale. For example, in Table 6.6, the user plans to sell 
4,000 bushels during January, which will leave 16,000 bushels for sale. The 
lower four shaded rows refer to financing decisions available to the user. The 
plan shown in Table 6.6 involves repaying $15,000 of operating debt in 
March, taking out a $17,000 long-term note in April, no additional short¬ 
term borrowing, and no inflows or outflows to savings. (Repayment of long¬ 
term notes is considered a fixed expenditure and is included in the 
expenditure section of the cash flow.) 

The table converts the relatively sterile cash flow preparation process into 
a dynamic decision-making framework. As the user alters planning market¬ 
ing or financial actions, the repercussions of those actions are immediately 
registered. Because many agricultural producers are concerned with cash 
flow issues, the ending cash balance line of Table 6.6 is a vivid anchor point 
for their decision making. The twelve-month nature of the monthly cash 
flow requires that the table be somewhat unwieldy. The five months shown 
are all that can be seen on the computer screen at one time. As the user 
moves to later months in the year, he or she can “scroll” to see the additional 
months. Both the column and row titles remain on the screen at all times, 
but this process is awkward. That problem is dictated by the decision 
maker’s choice of detail, however. For example, a quarterly version of the 
same statement would be much more user friendly if a particular user was 
comfortable with that level of information. 

LQANMON DSS 

The third DSS is also concerned with the cash flow position of the agricul¬ 
tural producer but from the perspective of the lender, not the borrower. Tra¬ 
ditionally, agricultural operating loans are made at the start of the calendar 
year, with fund disbursement and repayment occurring at several times dur¬ 
ing the year. Because of market and environmental uncertainties, however, 
the borrower’s actual financial position can be dramatically different at the 
end of the year than it was expected to be. For borrowers known to be In 
marginal circumstances at the time the loan is initiated, it is routine practice 
to require that the borrower’s actual cash position be reported to the lender 
monthly as the year progresses. Such reporting is a costly practice, and 
many borrowers resist this requirement. 
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As the year progresses, events may occur that markedly affect the finan¬ 
cial position of a loan officer’s clients. Droughts in the local region may 
reduce yield expectations, whereas droughts in competitor regions may raise 
price expectations. Political events, such as grain embargoes or announce¬ 
ments of new trade agreements, may substantially alter the market outlook. 
Although the loan officer has detailed financial projections data for each 
client, it is too costly to recalculate those projections manually for each bor¬ 
rower. The LOANMON DSS was created by a student with several years of 
experience as a loan officer. Its purpose is to allow the loan officer to alter a 
few key variables and observe the financial effect of those changes on the 
client portfolio. In particular, the DSS is designed to identify borrowers for 
whom the changed circumstance may imply difficulty in future loan repay¬ 
ment. Such forewarning would allow the borrower and the lender to con¬ 
sider contingency plans before the repayment problem became a crisis. 

At the time of the loan application, expected values for a wide range of 
variables are obtained. These include expectations of: 

• Prices and production for each commodity produced. 

• Amounts of other sources of income. 

• Cash operating expenses. 

• Required debt repayment. 

• Capital purchases. 

• Withdrawals for family living. 

These values are entered into the LOANMON DSS through input screens as 
for the previous two DSSs. At that time, the DSS is essentially a database of 
the loan officer’s portfolio. 

For each borrower, the loan officer determines a critical cash flow value at 
the time that the loan is made. This critical value represents a financial level 
at which the loan officer would feel the need to monitor that loan more 
closely. It does not necessarily mean that loan repayment is in jeopardy. 
Rather, it implies a financial level at which additional loan servicing may be 
required. This critical cash flow is a key to LOANMON’s structure. 

Understanding LOANMON may be aided by an example. Assume that 
market conditions have deteriorated since the start of the calendar when 
most loans were initiated and that the harvest price of corn has fallen from 
S2.50 to $2.00 per bushel. The loan officer would enter prices for all com¬ 
modities representing these new market expectations. Based on these expec¬ 
tations, LOANMON recalculates cash flow projections for each borrower. 
Note, however, that the market prices used for each borrower are not neces¬ 
sarily those just entered. Instead, each borrower’s original expected price is 
adjusted by the general change in market price that has occurred (a decline 
of 20 percent in this example). This allows individual differences in market- 
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ing ability, which were embedded in the original values, to be preserved but 
general changes in price level to be evaluated. 

LOANMON has the capability to produce two major output reports. 
These reports list the original budgeted cash flow, the cash flow currently 
estimated, the critical cash flow value, and the difference between the cur¬ 
rently estimated and the critical value cash flow. One of these reports will list 
all of the borrowers in the portfolio. The second lists only those borrowers 
on the watch list—those for whom the difference is negative between the 
currently estimated and the critical value cash flow. Table 6.7 presents an 
example of this second report for a hypothetical set of borrowers. In addi¬ 
tion to the borrower list, the report contains a listing of the price assump¬ 
tions used to prepare the analysis. These data can be invaluable in helping to 
explain the cash flow results to other officers of the financial institution or 
io the borrower. 

LOANMON illustrates the use of a DSS in an operational, structured de¬ 
cision situation. In fact, the data input can be accomplished by a clerk and 


Table 6.7 

Example of the Critical Value Cash Flow Report for LOANMON 


First Warmhearted Bank 
Report of Farm Borrowers 
Whose Projected Cash Flow is 
Less Than Critical Value 


5ase 

Cash Flow 
Budgeted 

Current 

Price 

Cash Flow 

Cash Flow 
Critical 
Value 

Difference 

Creenup. Fred 
Revere, Paul 
Sodbuster, C. 
White, York 

6,683 

20,560 

113,500 

25,842 

9,283 

(7,650) 

(23,660) 

(2,478) 

25,000 

60,000 

65,000 

30,000 

(15,717) 

(67,650) 

(88,660) 

(32,478) 


Price Assumptions 


Com, $/bu. 
Beans, §/bu. 
wheat, S_/bu. 
Cattle, $/cvt 
Hogs, $/cvt 


$2.37 
$5,32 
$3.64 
$53.20 
$41.25 
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the watch list report delivered weekly throughout the year. If a dramatic 
event occurs, the process can be initiated quickly. The LOANMON DSS is a 
classic illustration of filtering. Considerable data are needed to perform the 
necessary calculations. However, only a few key output variables are pre¬ 
sented to the decision maker and then only for borrowers for whom special 
attention may be needed. 


SUMMARY AND IMPLICATIONS 

This chapter addressed the potential for development of spreadsheet- 
based decision support systems. Drawing upon several years of experience in 
supervising student-generated DSS prototypes, it focused on two inter¬ 
related questions: 

1. Can useful DSSs, applied to nontrivial business problems, be developed by non- 

programmers using a general-purpose software package such as Lotus 1-2-3? 

2. Can such spreadsheet-based tools be developed so as to overcome psychologically 

based limitations of human decision making? 

In both cases, the evidence presented supports an affirmative response to 
these questions; however, important qualifications also are indicated. 

The prototype DSSs shown here indicated that useful programs can be 
developed to investigate real business issues. The menu-driven nature of 
these prototypes, as illustrated for the CAPP’s RFD example, results in pro¬ 
grams that are easy to use. This ease of use is very important for the “non¬ 
computer-oriented,” manager as well as in making the transactions costs of 
using the program much lower for the computer literate user. Spreadsheet- 
based DSSs, addressing realistic business problems, are not trivial undertak¬ 
ings. Although no formal reporting process has been in place, observation 
and discussion with the student developers suggest that forty to eighty hours 
of intense effort may be needed to reach the prototype stage outlined here, 
plus additional time for final modification and documentation. Although 
the payoff from use of these DSSs may be substantial, it is critically impor¬ 
tant that everyone involved in development understand the time costs poten¬ 
tially required. 

Explicitly addressing decision-making concepts in DSS development 
seems a worthwhile endeavor. Although everyone wants user-friendly pro¬ 
grams, we have little in the way of guidelines to use in creating such pro¬ 
grams. To this point, the emphasis has been on interface features, such as 
menus or means to reduce keystroke requirements. These options are impor¬ 
tant, but the truly user-friendly program must deliver the information 
needed for decision making in a concise and effective manner. As illustrated 
in the three example DSSs, tailoring of DSS, for either specific decision situ¬ 
ations or the needs of individual decision makers, is a major potential bene¬ 
fit of the spreadsheet-based approach. 
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A Spreadsheet System to Support 
the Decision-Making Process 


Ron Janssen 


Decision support systems (DSS) can be defined as “interactive computer- 
based systems that help decision makers utilize data and models to solve 
unstructured problems” (Gorry & Scott Morton 1971). Although this defini¬ 
tion is now more than twenty years old and although numerous competing 
definitions have been developed, this definition contains all the elements 
that are essential to the concept of decision support. “Interactive” and “com¬ 
puter-based” are essential features of DSS. By using a DSS, decision makers 
no longer communicate with models through an analyst but have direct ac¬ 
cess to available models and information. “Help the decision makers” high¬ 
lights the point that decision makers as individuals are supported. “Utilize 
data and models” emphasizes that support is given by processing infor¬ 
mation. 

Multiobjective decision support systems (MODSS) are a specific subcate¬ 
gory of DSS. An MODSS concentrates on the support of analysis/evalua- 
tion in a decision process (Figure 7.1) (Mintzberg, Raisinghani & Theoret 
1976). A MODSS includes formal methods such as multicriteria methods, 
graphic methods, and cost-benefit analysis combined with methods for 
problem definition and sensitivity analysis. In this way MODSS can support 
both the selection and the development phase of a decision process (Janssen 
1992). 
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Figure 7.1 

Support of the Selection Phase 



The cycle that supports the selection phase is shown in Figure 7.1. In the 
first round of this cycle, the results of analysis/evaluation enter either judg¬ 
ment evaluation/choice or bargaining evaluation/choice. Results of this 
first round are fed back into analysis/evaluation and may lead to changes in 
priorities and assumptions. By cycling repetitively, the influence of priorities 
and assumptions can be analyzed. Mintzberg, Raisinghani and Theoret 
(1976) qualify this type of cycle as a comprehension cycle. 

The feedback cycle supporting the development phase is shown in Figure 
7.2. In the first round of this cycle, the results of analysis/evaluation enter 
judgment evaluation/choice or bargaining evaluation/choice. If the results 
are unsatisfactory, which is to be expected in the first round, a feedback 
occurs to design (to alter existing alternatives or design new alternatives) or 
to search for new alternatives. 
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figure 7.2 

support of the Development Phase 



Cycling through the system supports elaboration of the problem: one idea 
ctivates a whole set of ideas. It also supports selection: identifying the key 
ssues of the problem. This may result in an adjustment of existing alterna¬ 
tes, the development of new alternatives, and the deletion of irrelevant al- 
srnatives. The purpose of this function is to obtain a more complete and 
dequate set of alternatives. 

The functions ascribed to a MODSS have resulted in the development of 
)EFINITE, an MODSS to support DEcisions with a FINITE set of alterna¬ 
tes in relation to a finite number of criteria. 1 The system can be used to 
upport a variety of environmental planning problems, ranging from siting 
he new high-speed train link from Brussels to Amsterdam to the selection of 
policy scheme to reduce carbon dioxide emissions. The system supports 
he whole decision process, from problem definition to report generation. 

HE DEFINITE PROGRAM 

DEFINITE aims at users with sufficient general education but without 
pecific experience in evaluation methods or computer programs and with 
me constraints. The five modules of the system are shown in Figure 7.3. 
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Figure 7.3 

DEFINITE Structure 


ArPr 


iskiOQ 


1 


^ B: Problem, presentation; ^ 



^ C: Problem evaluation; 

1 


C ^SensitcvilyanalysSs: ) 



^ E: Report generation J 


Problem definition results in an effects table representing the decision 
problem. Problem presentation results in various presentations of the effects 
table. The effects table can be presented as an appraisal table, a graph, a 
scatter diagram, or a cost-benefit sheet. Problem evaluation generates 
a complete or incomplete ranking of the alternatives, the best alternative, or 
a set of acceptable alternatives. The presence of multiple dimensions, some¬ 
times even of different measurement scales, is handled through multicriteria 
analysis or cost-benefit analysis. The sensitivities of the results of problem 
evaluation to uncertainties in scores, weights, and prices, together with the 
sensitivity of the ranking to the evaluation method used, are analyzed in 
module D, sensitivity analysis. All results are combined into an evaluation 
report in module E, report generation. 

Each module contains a variety of procedures to perform these functions. 
In the following section a selection of these procedures is described. A full 
description of the DEFINITE program can be found in Janssen and van 
Herwijnen (1992). The program can be described according to five topics: 

1. The completeness of the effects table. An important difficulty in problem defini¬ 
tion is to reach a complete effects table. Two procedures are included to generate a 
set of alternatives that is as complete as possible and one procedure to support the 
selection of the evaluation criteria. 

2. Possibility of low information level. Effect scores are not always available as hard 
numbers. To be able to include all information, the program accepts low informa¬ 
tion scales such as ordinal, binary, and nominal scales. 

3. Creativity in problem formulation. The user Is stimulated and supported to create 
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new alternatives or new combinations of alternatives. Since procedures are avail¬ 
able to reduce the number of alternatives at a later stage, the user is not restrained 
from inventing infeasible alternatives. 

4. Presentation versus aggregation. In problems with a small number of alternatives 
or criteria, an adequate presentation can be sufficient to evaluate the alternatives. 
In these cases, the alternatives are ranked according to the overall impression pro¬ 
vided by the presentation. If an adequate presentation is available, there is no 
need to use a formal decision rule to aggregate the scores. 

5. Negotiations . In module D, the sensitivity of all chosen values and methods can 
be examined. Certainty intervals can be calculated for weights and scores so that 
within such an interval, the ranking of alternatives is not sensitive to changes in 
scores or weights. This is useful in negotiations if there are different opinions on 
scores and priorities. 


PROBLEM DEFINITION 

The purpose of module A, problem definition, is to obtain an effects table 
that represents the decision problem. All procedures of this module are 
shown in Figure 7.4. 



Figure 7.4 

Procedures Model A: Problem Definition 
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Completeness 

An important difficulty in problem definition is reaching a complete ef¬ 
fects table. Completeness of the set of alternatives may be achieved by using 
procedure A3 in Figure 7.4. The user is requested to identify a number of 
aspects of the problem at hand and to specify, for each aspect, options for 
the solution of the problem. Furthermore, the user specifies which combina¬ 
tions of options are not feasible. The combination procedure now generates 
all possible alternatives. 

Example 1 

The seaside resort of Deil aan Zee is separated from the main urban area 
by the river Traach. The problem is how to provide a connection across this 
river. Table 7.1 shows three aspects to this problem and two or three possible 
solutions for each aspect. 

Table 7.1 

Input to the Combination Procedure 


aspects 

solutions 



Means of transport 

Train 

Car 


Type of connection 

Ferry 

Tunnel 

Bridge 

Capacity (persons/day) 

10,000 

100,000 



The user subsequently specifies that the combination Train and Ferry and 
the combination of Ferry and a capacity of 100,000 persons/day are not 
feasible. The procedure generates nine alternatives as shown in Table 7.2. 

Table 7.2 

Output of the Combination Procedure 


Alt 1 

Alt 2 

Alt 3 

Alt 4 

Alt 5 

Alt 6 

Alt 7 

Alt 8 

Alt 9 

Train 

Train 

Train 

Train 

Car 

Car 

Car 

Car 

Car 

Tunnel 

Tunnel 

Bridge 

Bridge 

Ferry 

Tunnel 

Tunnel 

Bridge 

Bridge 

10,000 

100,000 

10,000 

100,000 

10,000 

10,000 

100,000 

10,000 

100,000 


A complete set of alternatives can also be achieved by aggregating all pos¬ 
sible combinations of elements into alternatives. This is performed by proce¬ 
dure A4. The user specifies elements and combination rules. The procedure 
generates all combinations of elements that comply with these combination 
rules. 
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Example 2 

To secure the water supply for the seaside resort of Deil aan Zee a mini¬ 
mum of 10,000 mVyear needs to be extracted from groundwater stocks. 
Table 7.3 shows that four wells are available in the dune area nearby. These 
wells (the elements) have a capacity of 3,000, 5,000, 2,000, and 6,000 m 3 / 
year. 

Table 7.3 

Input to the Aggregation Procedure 

Element 1 Element 2 Element 3 Element 4 


Capacity nxVyear 3,000 5,000 2,000 6,000 


Land use nr 200 400 200 700 

Groundwater cm -5 -10 -4 -15 

Combination rule: each alternative must produce at least 10,000 mVyear 
and no more than 12,000 mVyear. This results in the following alternatives: 

Alternative 1 = Element 1 + Element 2 + Element 3 
Alternative 2 = Element 1 + Element 3 + Element 4 
Alternative 3 = Element 2 + Element 4 

The resulting effects table as shown in Table 7.4 is derived by summing for 
each alternative the scores of the elements (the columns of Table 7.3) in¬ 
cluded in the alternatives. 


Table 7.4 

Output of the Aggregation Procedure 



Ml 1 

Alt. 2 

Alt. 3 

Capacity oxVyear 

10,000 

11,000 

11,000 

Laud use m 2 

800 

1,100 

1,100 

GroiflMtwater cm 

-19 

-24 

-25 
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Low Information Level 

Effect scores are not always available as hard numbers. To be able to in¬ 
clude all information DEFINITE accepts low information scales such as or¬ 
dinal, binary, and nominal scales (see Table 7.5). 

Example 3 

Traffic congestion occurs on a road between Deil aan Zee and the nearby 
city of Nijverhoeck. Three alternatives are available to solve this problem: a 
four-lane highway, a new road combined with special bus lanes, or a new 
train connection. The expected effects of the three alternatives are listed in 
Table 7.5. 

Table 7.5 

Improve Transportation to Nijverhoeck 

criteria units Highway Road/bus Train 


Costs 

min gld. 

200 

250 

400 

Travel time 

—/+++ 

+++ 

++ 

+ 

Capacity 

mln/km/year 

20 

30 

40 

NO* emissions 

ton/year 

1,000 

750 

100 

Landscape 

—/+++ 

— 

— 

- 


PROBLEM PRESENTATION 

The purpose of module B is to present the effects table in a form that 
supports comparison of the alternatives without applying a formal decision 
rule. All procedures in module B are shown in Figure 7.5. 


Figure 7.5 

Procedures Module B: Problem Presentation 
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Presentation versus Aggregation 

Presentation can replace aggregation in problems with a small number of 
alternatives or criteria. In these cases, the overall impression provided by the 
presentation replaces the scores calculated by decision rules. 

Example 4 

A graphic presentation of Table 7.5 is shown in Figure 7.6. For each crite¬ 
rion, the highest bar indicates the best alternative. After the criteria are 
ordered, in descending priority, the alternatives can be ranked visually (Fig¬ 
ure 7.7). 2 


Figure 7.6 

Graphic Presentation of an Effects Table 

Cests 

1r«««t 1 in< 
ity 

;«ft* hdm 

Mi^huay Tr»in 



Figure 7.7 

Graphic Presentation of a Sorted Effects Table 


Capacity 
Exi*si«n« KDx 
<•*** 

Travel tixt 
Landscape 



Highway 


PROBLEM EVALUATION 

This module contains two main categories of procedures: monetary evalu¬ 
ation methods and multicriteria methods. All methods result in a ranking of 
the alternatives by aggregating the effects table according to the decision 
rules assumed. All procedures in module C are shown in Figure 7.8. 


Low Information Level 

Multicriteria analysis requires information on priorities. A decision 
maker can usually express opinions on priorities only in a qualitative form. 
DEFINITE contains various procedures to transform this low-level infor¬ 
mation into quantitative weights as required by the various multicriteria 
methods. 





Figure 7.8 

Procedures Module C: Problem Evaluation 
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The expected value method, for example, asks the user to order the effects 
in descending priority order. Each weight is calculated as the expected value 
of all values conforming to the rank order of that weight. Another method, 
the analytical hierarchy process (AHP), requires much more information. In 
this method, the user is not only asked to compare each pair of criteria but 
also to express whether one criterion is more important, strongly more im¬ 
portant, and so forth, than the other criterion. The weights are calculated as 
the eigenvalues of the resulting matrix of pairwise comparisons. Other 
methods for dealing with qualitative information or priorities are the ex¬ 
treme value and the random value method. 

Example 5 

To calculate the expected values of the weights the user has specified that 
capacity is as important as NO x emissions, NO x emissions are more impor¬ 
tant than costs, and so on. As described above, to calculate the eigenvalues 
information on all pairs of criteria is required. In this example capacity is 
equally important as NO x emissions, capacity is moderately more important 
than costs, capacity is strongly more important than travel time, and so on. 
The results of both methods are shown in Table 7.6. 


Table 7.6 

Weights according to the Expected Value Method and the Eigenvalue 


ranking 

Expected Value 

Eigenvalue 

1. Capacity 

0.357 

0.379 

NO* emissions 

0.357 

0.379 

2. Costs 

0.157 

0.140 

3. Travel time 

0.065 

0.051 

Landscape 

0.065 

0.051 


DEFINITE contains five multicriteria methods to transform the effects 
table in combination with the weights into a ranking of the alternatives: 

• The weighted summation method derives the ranking from the weighted sum of 
standardized effect scores; 

• The electre method is based on graphs derived from pairwise comparisons of all 
alternatives; 
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• The regime method is specially designed to handle qualitative or partly qualitative 
effects tables. The method is based on partitioning the set of values in accordance 
with the ordinal effects scores; 

• The expected value method derives the ranking from the weighted sum of the ex¬ 
pected value of the effect scores; 

• The evamix method ranks the alternatives according to a combination of a domi¬ 
nance index calculated from the qualitative scores, and a dominance index calcu¬ 
lated from the quantitative scores. 

The results of the regime and expected value methods are presented in Tables 
7.7 and 7.8. 

Table 7.7 

Results of the Regime Method 


WEIGHTS 

rank 

RANKING 

rank 

score 

1. Capacity 

1. Train 

1.00 

NO, emissions 

2. Road/bus 

0.46 

2. Costs 

3. Highway 

0.04 

3. Travel time 



Lancfecape 




Table 7.8 

Results of the Expected Value Method 

WEIGHTS 


RANKING 


rank 

score 

rank 

score 

1. Capacity 

0.357 

1. Train 

0.82 

NO, emissions 

0.357 

2. Road/bus 

050 

2. Costs 

0.157 

3. Highway 

027 

3. Travel time 

0.065 



Landsc^e 

0.065 
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These tables show that both methods result in the same ranking of the 
three transportation alternatives. To apply the regime method only qualita¬ 
tive information on weights was required. To apply the expected value 
method this qualitative information was first transformed into quantitative 
weights. 

SENSITIVITY ANALYSIS 

The sensitivities of the rankings obtained in module C to changes in eval¬ 
uation methods, effect scores, and weights are analyzed in module D (see 
Figure 7.9). The module contains procedures to: 

• assess the sensitivity of the ranking to the evaluation method applied (method un¬ 
certainty); 

• assess the influence of uncertainties in scores and weights on the ranking of the 
alternatives (effects and weights uncertainty); and 

• determine the intervals within which the rank order of two alternatives is insensi¬ 
tive to changes in scores or weights (effect and weight intervals). 

Special search procedures are developed for determining these intervals for 
all evaluation methods included in the system. 



Figure 7.9 

Procedures Module D: Sensitivity Analysis 
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Negotiations 

Score, weight, and price intervals can be useful in negotiations when par¬ 
ticipants hold different views on certain scores or priorities. 

Example 6 

As shown in Table 7.7, the regime method ranks alternative train before 
road/bus and highway. In Figure 7.10, certainty intervals are calculated to 
test the stability of the highest-ranked alternative with respect to the ex¬ 
pected capacity of the alternatives highway and road/bus. The difference 
between the start value and the reversal value is considerably smaller in the 
comparison between the alternatives train and road/bus (Figure 7.11) than 
in the comparison between alternatives train and highway. 

Figure 7.10 

An Effect Interval for the Score of Capacity' of Alternative Highway 

Train > Highway Highway 

> Train 

i- 1 - 1-► 

o 20 43 

start reversal 


Figure 7.11 

An Effect Interval for the Score of Capacity of Alternative Road/Bus 


Train > Road/bus 


Road/bus 
> Train 


30 

start 


41 

reversal 


Example 7 

The same can be done in procedure D4 where DEFINITE calculates cer¬ 
tainty intervals for weights. The user specifies the weight to be tested and 
two alternatives for comparison. The result of this procedure is presented in 
Figure 7.12. This figure shows that the weight assigned to the criterion travel 
time needs to be increased considerably to change the ranking order of train 
and highway as calculated with the expected value method. 
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Figure 7.12 

A Weight Interval for Criterion Travel Time 

Train > Highway Highway > Train 

H-1-1 

0 0.065 0.608 1 

start reversal 

Only the relative weight of criterion travel time was changed in calculating 
the weight intervals in Figure 7.12; the ratios of all other weights were held 
constant. If all weights are allowed to change freely, it becomes clear how 
sensitive the ranking is to overall changes of the weights. 

Table 7.9 shows the weight set with the smallest Euclidean distance from 
the original weight set that causes a rank reversal of alternatives train and 
highway. This table shows that the nearest weight combination that results in 
a rank reversal differs considerably from the original set of weights. 

Table 7.9 

A Weight Combination with Rank Reversal: Expected Value Method 


original ranking 


Capacity 

0.357 

1. Train 

NO* emissions 

0.357 

2. Road/bus 

Costs 

0.157 

3. Highway 

Travel time 

0.065 


Landscape 

0.065 



reversed ranking (distance 0.316) 


Costs 

0.347 

1. Road/bus 

Capacity 

0.203 

2. Highway 

NO* emissions 

0.192 

Train 

Travel time 

0.177 


Landscape 

0.081 



CONCLUSIONS 

The use of DEFINITE to support the decision process has clear advan¬ 
tages: 

• It supports the decision making as a cyclical process. 

• It enables the decision maker to incorporate all relevant information in the deci¬ 
sion; it counteracts the unsatisfactory practice of limiting the information in¬ 
cluded to the number of issues that can be handled by a single individual. 

• It stimulates learning by doing. 

• It can focus debate in decision making by negotiation. 
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DEFINITE has a bias toward rational, normative decision making. With 
the exception of the graphic presentation, all results are produced using a 
formal decision rule. DEFINITE will not be popular with decision makers 
who seek other, perhaps more subjective decision procedures. In further de¬ 
velopment of MODSS, it is essential to put more emphasis on the behavioral 
aspects of the use of these systems (see also Yu 1990; Timmermans 1991; 
Ylek, Timmermans & Otten 1991). 

NOTES 

1. The development of DEFINITE was commissioned by the policy analysis de¬ 
partment of the Ministry of Finance, the Hague. The Dutch version of DEFINITE 
was developed in cooperation with this department. DEFINITE is available from 
Kluwer Academic Publishers, Dordrecht (Janssen & van Herwijnen 1992). The soft¬ 
ware of the DEFINITE system was written by M. van Herwijnen. 

2. If prompted, DEFINITE performs the sorting of the columns automatically 
using the expected value method in combination with weighted summation. 
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Program Evaluation via 
Function-based Expert Systems 

Paul J. Hoffman 


The term expert systems software sounds more esoteric and high-tech than it 
ought to be. In reality, the concept is simple. The objective is to create com¬ 
puter software that will enable users to emulate experts —that is, to have the 
computer make decisions about alternatives in the same way that knowl¬ 
edgeable experts would make those decisions. 

Perhaps the most familiar example of expert systems software is an airline 
reservation system. Enter a point of departure, a date, and a destination 
from the keyboard, and a good airline reservation system will display the 
flights that have seating available, along with their routes and type of ser¬ 
vice. A less familiar example is a system for the evaluation of research pro¬ 
posals. Select a proposal from the software’s data files; then select a 
committee from the software’s expert system directory. The proposal is auto¬ 
matically evaluated with respect to the standards and objectives of the com¬ 
mittee and is immediately displayed to the user, ranked in relation to the 
other proposals on each of a number of key factors. 

Program evaluation is exactly analogous to proposal evaluation. The only 
difference is that the “committee” whose expertise is available in the com¬ 
puter’s files is in fact composed of stakeholders with different frames of ref¬ 
erence and different levels of responsibility for decision making. 

The origin of such systems is clearly with an expert or group of experts. 
For airline reservations, the group of experts are available to the software 
developers to help make decisions about routes and to assist in the formula¬ 
tion of algorithms. One such algorithm is availability (what number of extra 
passengers do you book when the space on a flight is 100 percent sold, on 
the expectation that a given number of the passengers will be no-shows?). 
For program evaluation, the knowledge and experience of the stakeholders 
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(about the decision problem, not about computers) is the key to effective, 
precise, and well-documented decision making, and the algorithms needed 
have to do with modeling the personal, intuitive judgmental processes of the 
stakeholders, who in real life must make trade-offs between various out¬ 
comes and objectives. 

Expert systems can be employed in a wide range of activities, and to con¬ 
siderable advantage: business decisions, policy studies, medicine, sentencing 
guidelines, personnel selection, and other fields of endeavor. Their main ad¬ 
vantage is an obvious one: organizations will be able to utilize expertise in 
their decisions, even if the experts are physically unavailable at the time. If 
the expert systems have been properly engineered, such organizations will 
have use of a completely reliable tool for their decisions — a tool with inter¬ 
nal logic that presumably has been subjected to rigid testing to ensure that 
automated decisions properly represent the decisions that would in fact have 
been made by the organization’s experts, had they been available. 

This chapter illustrates the manner in which expert systems can be so em¬ 
ployed, explains how they operate, and illustrates their role in decision sup¬ 
port. Discussion will be limited fairly exclusively to function-based expert 
systems (Hoffman 1985), as opposed to rule-based expert systems (Harmon 
1985). 

RULE-BASED EXPERT SYSTEMS 

Rule-based expert systems evaluate information by means of a set of rules 
that experts believe are Important to the decision and that system developers 
build Into the software of their system. These are, for the most part, Bool¬ 
ean expressions, that is, of the form: 

If A and B and not (C or D) then X. 

For example, consider the evaluation of candidates for a postdoctoral fel¬ 
lowship. The above rule might take the form: 

If the candidate scores above the 99th percentile on the Graduate Record Exami¬ 
nation and the candidate has strong recommendations from the faculty and the can¬ 
didate is not a biology or a social science major then 

Assign the candidate a B-f rating. 

A single rule-based expert system typically is composed of a very large 
collection of rules; if the number exceeds 100 to 150, as it often does, the 
rules are structured so that the information required by certain rules is pro¬ 
vided by the application of other rule sets to the information base. The 
mechanism that controls the evaluation of rule sets is often referred to as the 
inference engine. When a subset of information is entered by the user, the 
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computer-based processes set in motion include testing individual rules, 
drawing inferences from the available information, and identifying the need 
for new information. The range of complexity and diversity of rule-based 
expert system tools is considerable. The objective is to reach conclusions 
among alternatives, based on the information base, the knowledge base, in¬ 
cluding the rule set, and new information provided by the user. 

There are a number of problems associated with Boolean rule-based sys¬ 
tems, among them, obfuscation, insufficiency, capitalization on chance, va¬ 
lidity, and relativity. 

Obfuscation occurs as an inevitably natural by-product of the develop¬ 
ment of rule-based systems. It becomes apparent when users attempt to un¬ 
derstand the logical process that underlies the decisions made by the system. 
Because of the sheer complexity of typical rule sets and the algorithms that 
control the order of their application, few, if any, users will develop a clear 
understanding of the basis of machine-made decisions. 

Insufficiency refers to errors in the evaluation of alternatives that result 
from the use of machine-derived inferences in the absence of needed data. 
Given a degree of uncertainty in the quality of the inferences, the problem 
occurs because rules may assume the presence of certain information re¬ 
quired by the rule, and the absence of the information can prevent the rule 
from being applied, unless the information is derived from other informa¬ 
tion sources, evaluated by other rules. The need for derived information 
may be trivial or extensive. The more extensive the need is, the more uncer¬ 
tainty is introduced into the evaluation. Thus, the accuracy of evaluations 
may be high or low, depending on the sufficiency of information available. 
The user will not likely know the seriousness of this effect. 

Capitalization on chance occurs in two principal ways: when the inference 
engine begins processing rules in a search of relationships within the data¬ 
base and, in the course of development of the software, when the developer 
Iteratively tests and corrects the program so as to minimize the frequency 
and seriousness of erroneous results. In both cases, the accuracy of results is 
inflated because degrees of freedom are “used up” in the course of fitting 
rule structures to the data. This problem is analogous to the well-known 
shrinkage problem (the overfitting problem) in regression analysis (Mc- 
Nemar 1949). 

Validity refers to the accuracy of the decision model. When applied to 
expert systems, accuracy has two connotations: it may refer to the ability of 
the expert system to make correct decisions and to the fidelity of the expert 
system, that is, the extent to which the decisions made by the expert system 
are the same as the decisions that would have been made by a reliable expert 
or group of experts on whom the expert system is based. Users of rule-based 
expert systems should attempt to ensure that at least one of these two tests of 
validity has been met before trusting the results. 

Relativity refers to the manner in which the expertise of a group or com- 
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mittee is distributed among the different components of the problem. Given 
that any alternative is evaluated on the basis of objective data, it is neverthe¬ 
less the case that the same set of objective data can and probably should be 
evaluated differently by different experts. That is, experts will differ in the 
manner in which they wish to interpret objective data, perhaps believing 
that certain outcome variables should be given more weight than others. 
And some experts will be properly seen as more knowledgeable or more in¬ 
sightful than others for certain components of the decision problem but not 
for other components, as a result of which the viewpoints (and personal 
standards) of different experts should be given more weight in proportion to 
their expertise. Rule-based expert systems almost universally neglect this 
problem. 

Herein lies an important principle, which applies as well to everyday deci¬ 
sion making as to sophisticated, computer-based expert systems: the evalua¬ 
tion of alternatives is, and should be, relative to the standards and values of 
the expen. Two important requirements follow from the principle: 

1. Expert systems should be constructed so as to permit the allocation of expertise 
among various components of the problem, according to the talents and responsi¬ 
bilities of the experts. 

2. When utilized as decision support systems, expert systems should clarify for the 
user the exact manner in which expertise has been distributed among the various 
components of the problem. 


FUNCTION-BASED EXPERT SYSTEMS 

Function-based expert systems use algebraic functions to emulate the in¬ 
tuitive mental processes of experts engaged in decision making. A simple 
example is an expert system for graduate admissions based on the appli¬ 
cant’s undergraduate grade point average (GPA), Scholastic Aptitude Test 
(SAT) scores, and a measure of the quality of the undergraduate program 
(Dawes 1971). In this example, it was possible to express (with considerable 
validity) the decisions made by admissions committees as simple weighted 
functions of the three variables: 

J = aX + a 2 Y + a 3 Z + <2 0 

where 

J = the committee judgment of each applicant, 

X, Y, and Z = the attributes GPA, SAT, and quality of the under¬ 
graduate program, and 

% a i> a 2 > and a 3 = parameter estimates of a linear model that opti¬ 
mizes the least-squared deviations of the model’s predictions 
from the committee’s judgments. 
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In the study cited, Dawes found that the model of the committee was not 
only more reliable than the committee itself but also that the predictions of 
graduate success based on the model were more valid than were the predic¬ 
tions made by the committee. The study is a classic illustration of the boot¬ 
strapping effect (Hogarth 1980; Ebert & Kruse 1978), demonstrating the 
superiority of simple models over the more fallible judgments of indi¬ 
viduals. 


Hierarchical Structure 

The graduate admissions example is atypical of personnel selection deci¬ 
sions in industry and government, wherein each applicant must be evaluated 
on a score or more variables. Indeed, the example is atypical of program 
evaluations and policy evaluations for the same reason. To do justice to eval¬ 
uation, one must examine a multitude of information from a variety of 
sources, and not all of the information is initially in objective form. Can 
simple regression analysis be used for these other classes of problems? 

The answer is a qualified “no.” In Dawes’s study, data were available from 
a moderately large (N = 77 to 93) sample of students. In addition, a very 
small number of variables was considered, and a surprisingly simple linear 
model was deemed a priori to be sufficient. Therefore, the residual degrees 
of freedom were large enough to permit reasonably accurate estimation of 
the parameters of the model. In a typical program evaluation study, how¬ 
ever, just the opposite conditions apply. Typically, the sample size (for exam¬ 
ple, of competing programs or interventions being evaluated) is small, the 
number of dependent variables is typically large, and a linear model might 
very well not apply. Therefore, regression analyses under these conditions 
are bound to yield poor or misleading parameter estimates. 

To deal with these problems, function-based expert systems give careful 
attention to the hierarchical structure of dependent variables. These varia¬ 
bles, which are referred to as attributes , are grouped into meaningful clus¬ 
ters, and the clusters are assigned names. Each cluster is referred to as a 
concept, to denote the fact that it exists at a higher level of abstraction 
(Hayakawa 1947) than its defining attributes. 

To illustrate the distinction between attributes and concepts, consider the 
task of a National Institutes of Health (NIH) study section on cardiovascu¬ 
lar disease and stress, evaluating competing research proposals. 1 Were the 
committee to jot down the factors that should be considered in evaluating 
the proposals, an unstructured list might be like the one shown in Fig¬ 
ure 8.1. 

In a function-based expert system, the organization’s experts and the de¬ 
veloper work to cluster the attributes into an organized hierarchy that is 
uniquely meaningful for the evaluation problem. For example, items 1, 7, 9, 
11, and 12 form a related set of attributes in the sense that each is a compo- 
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Figure 8.1 

Attributes for the Evaluation of Proposals 

1. Knowledge of the literature 

2. Competence of investigators 

3. Library resources available 

4. Seriousness of Public Health problem to be investigated. 

5. First year costs 

6. Long-term costs 

7. Analytic methodology 

8. No. of lives improved 
9- Salience of the problem 

10. Accessibility of a clinic 
11- Appropriateness of hypotheses 

12. Design Methodology 

13. Probability of project success 

14. No. of publications of research team 

15. Quality of publications of team 

16. Likelihood that project will engender public support 

17. Laboratory space available 

18. Computer resources available 

19. Support-staff adequacy 

Note: The list of attributes is undifferentiated. It lacks organization and coherence. 


nent of the scientific merit of any program or proposal. The label “scientific 
merit” was chosen by the committee, which by then understood clearly that 
it is defined by its subordinate attributes. If other attributes are considered 
essential to the meaning of the concept, they are added at this time. 

Further study of the attributes of Figure 8.1 reveals additional insights. 
Items 5 and 6 have to do with economics. Attributes 4, 8, 13, and 16 have 
to do with the public health impact of the study. An example of a completely 
structured hierarchy for the research proposal problem is shown in 
Figure 8.2. 

It is seldom difficult to work through the details of constructing problem 
hierarchies. In Figure 8.2, we have been able to group nineteen original de¬ 
pendent variables, or ratings, into five concepts: scientific merit, team com¬ 
petence, facilities, economics, and public health impact. 

While the first problem for function-based expert systems development is 
to conceptualize the problem dimensions hierarchically, the second problem 
is to generate algebraic functions for each of the concepts— functions that 
map attribute values into measures of the concepts in a manner that emu¬ 
lates the intuitive way in which experts would actually evaluate the concept, 
given the attribute set. This problem is solved by modeling the intuitive 
judgmental processes of experts. 

EXAMPLE-BASED EXPERT SYSTEM DEVELOPMENT 

The solution to this problem has its roots in paramorphic representation 
(Hoffman 1960), modified according to a particular set of specifications for 
the generation of hypothetical examples, to which experts respond with their 
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Figure 8.2 

Same Attributes, Structured Hierarchically 

OVERALL MERIT OF PROPOSAL 


1 SCIENTIFIC MERIT 
Knowledge of literature 
Salience of problem addressed 
Appropriateness of hypotheses 
Design methodology 
Analytic methodololgy 

2 TEAM COMPETENCE 
Training and credentials of the 

principal investigators 
Publications (quantity) of the 
principal investigators 
Publications (quality) of the 
principal investigators 
Support staff adequacy 

Note: They are essentially those shown in 
into more meaningful concepts. 


3 FACILITIES 
Library space available 
Computing resources available 
Laboratory space availability 
Accessibility/adequacy of clinic 


4 ECONOMICS 
Short-term costs 
Long-term costs 


5 PUBLIC HEALTH IMPACT 
Probability of project success 
Seriousness of Public Health problem 
Number of lives improved 
Impact on public support 

Figure 8.1, but they are now grouped hierarchically 


best judgments. The process is not unlike the well-known policy-capturing 
methodology described extensively by others (Slovic & Lichtenstein 1971, 
Hammond, McClelland & Mumpower 1980), except that the new features 
incorporated into EXPERT87 software overcome certain heretofore trouble¬ 
some aspects of policy capturing. The new methodology abandons the tradi¬ 
tional emphasis on empirical regression analysis in favor of hierarchically 
ordered design matrices for the generation of examples. A simple prototype 
of the application of these matrices can be found in Hoffman, Slovic, and 
Rorer (1968). 

The EXPERT87 approach incorporates the following new features: 

1. The attribute values used in the examples displayed to the experts are automati¬ 
cally created as optimally efficient, orthogonal, integer-valued sets. 

2. The models that are to become components of the expert systems may be either 
linear or nonlinear. 

3. The hierarchical structuring of attributes is such as to minimize the cognitive limi¬ 
tations in human information processing capacity (Miller 1956). 

4. The parameters of the models are unambiguous estimates of the independent ef¬ 
fects of the attributes and their nonlinear components. 

5. Absolute scaling is incorporated, enabling evaluations of alternatives to be inter¬ 
preted in terms of the standards of the experts. 

6. The modeling process is consistent with the hierarchical structure of attributes 
and concepts. 
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The end result of the expert system development phase is a set of expert 
systems for each concept of the evaluation problem— expert systems that 
may be used to evaluate any alternative with respect to its corresponding 
concept, rank all alternatives on any concept of interest, and evaluate any 
selected alternative in an absolute sense. 

Figure 8.3 is a computer-generated display that illustrates an important 
phase of function-based expert system development. The display is one of 
twenty-five examples presented to a stakeholder or committee member for 
judgment. The attributes displayed are those embodied in the first concept 
of the hierarchy, scientific merit. The expert interacts with the computer¬ 
generated examples at a terminal, entering a judgment on a 100-point scale 
in response to each example. In this instance, the expert has assigned a rat¬ 
ing of 46 to the profile (shown in the lower left corner of the figure). This 
number indicates the expert’s judgment of the scientific merit of a proposal, 
given the attribute values shown. 

The precise number of examples displayed is under control of the 
program. The number varies from concept to concept, because different 
concepts are defined by different numbers of attributes, requiring the esti¬ 
mation of different numbers of parameters. An algorithm within 
the program defines the attribute values for each example set, as well as the 


Figure 8.3 
A Sample Profile 



ENTER ¥001 ASSESSMENT OF SCIENTIFIC MERIT 

ENTER IT AS A 2-BIG IT INTEGER C e.g., 36 , 07 , 93 ) 



Esc CONCEPT MENU M Fl ABORT j F2 HELP 


ffsasEfi'Ea 


| ? £or help 


I 


F4 DEF1NTI0NS 


Sole: Experts (committee members or stakeholders) enter judgments of the concept repre¬ 
sented by the configuration of attribute scores. These hypothetical examples and the re¬ 
sponses of experts comprise the data for completely automatic development of expert 


systems. 
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number of examples required, so as to optimize accuracy of parameter esti¬ 
mation while minimizing the total number of examples required. 

Responses of experts to example sets provide the data required to generate 
the parameters of expert systems. This computational process is initiated im¬ 
mediately following the entry of the last judgment in a given example set, 
whereupon information about the expert’s intuitive judgmental processes is 
immediately displayed on-screen. 

An illustration of the parameters of an expert system is depicted in Figure 
8.4. An expert, Albert, has interacted with a set of examples, on each of 
which are displayed committee ratings of the attribute values of a hypotheti¬ 
cal proposal. Figure 8.2 depicts the attributes of one of the concepts of Al¬ 
bert’s expert system. Also shown are some of the parameter estimates of the 
model. These are listed as component weights, weights that can map the set 
of attribute values into a measure of the concept, in this case, team compe¬ 
tence. The mapping emulates the way in which Albert would intuitively re¬ 
spond to an actual proposal. That is, when the parameters of Albert’s expen 
system are applied to the attribute values of an actual proposal, the resulting 
polynomial expression reflects Albert’s personal standards and orientation 
with respect to the relative importance of the attributes. The component 
weights shown in the figure indicate that Albert gives significantly large 
weight to the investigators’ credentials and their publication quality. He 
gives little weight to support staff adequacy. 

In the figure, note the significant nonlinear (concave) effect correspond¬ 
ing to publication quantity. For Albert, this means that numbers of publica¬ 
tions of a research team signifies greater team competence and that the 

Figure 8.4 

Some Parameters of an Expert System 

SELECTED CONCEPT * TEAM COMPETENCE 
CANDIDATE CLASS = PROPOSAL, NIH/NCI 
EXPERT: (A) ALBERTI 

PROPOSAL. WTS 10—26—1986 09:59:22 

FIDELITY = 97.1 


RELATIVE WEIGHTS ANALYSIS 

COMPONENT WEIGHTS 



ATTRIBUTE 

LINEAR 

NON-LIN 

TOTAL 

SHAPE 

PI TRAINING & CREDENTIALS 
PI PUBLICATION QUANTITY 

PI PUBLICATION QUALITY 
SUPPORT STAFF ADEQUACY 

31.57% 

10.80% 

29.91% 

9.14% 

1.86% 

11.15% 

3.72% 

-1.86% 

33.43% 

21.95% 

33.63% 

11.00% 

PCS. LINEAR 
PCS.CONCAVE 
PCS. LINEAR 
PCS.LINEAR 


TOTAL 

VARIANCE EXPLAINED 

81.42% 

88.21% 

18.58% 

6.11% 

100.00% 

94.32% 



Note: The entries represent the relative importance of each attribute, as derived by EX¬ 
PERTS?. The relative strength of linear and non-linear effects is shown, as is the toun 
contribution of each, attribute to the evaluation of the concept. The fidelity index &P 


pears at the top of the figure. 
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incremental effect diminishes as the number of publications becomes large. 
Albert wants investigators to have at least a few quality publications to their 
credit, but beyond this, other factors come into play. Expert systems for the 
remaining concepts are derived in similar fashion, and their meaning can be 
similarly explicated. 

A third problem for the expert system development tool is that of generat¬ 
ing an algebraic function for mapping the derived concept scores into overall 
measures of the alternatives, thereby enabling the evaluation of alternatives 
with respect to their overall efficacy. This phase of development requires a 
slightly different kind of policy-capturing task, one whose hypothetical ex¬ 
ample sets depict alternatives with associated scores on each of the concepts 
rather than depicting concepts with associated scores on their attributes. 
The expert system that results from this process becomes a tool for evalua¬ 
tion of the overall efficacy of the alternatives, a process that, in the EX¬ 
PERTS? environment, is carried out automatically. 

FEEDBACK OF EXPERTISE 

Upon completion of these phases of expert system development, the user’s 
files contain a set of expert systems for each expert. This can be portrayed as 
a two-way table, where rows represent experts, columns represent concepts, 
and each individual cell represents an expert system that maps a subset of 
attributes into a measure of the concept in a manner emulating the expert. 
In the EXPERTS? environment, such a table is referred to as an expert sys¬ 
tem directory. It would appear, on-screen, as in Figure 8.5. 

If an expert has completed the judgments of an example set, an asterisk is 
displayed in the corresponding cell of Figure 8.5. If the user has decided to 
utilize a particular expert for evaluation of a selected concept, the letter P 
(patched) is shown in the cell. The configuration of patched expert systems 
represents the manner in which the user has decided to allocate the expertise 
of the stakeholders to the concepts of the evaluation problem. 

Should the user elect to patch more than one expert to a given concept, so 
as to alow their collective expertise to be emulated, this can also be accom¬ 
plished. What rationale can be used for aggregating expertise over stake¬ 
holders, or for choosing to accept or to reject the expertise of individual 
stakeholders? This question is discussed in the following paragraphs. 

Within the operation environment of the software, the user is able to ac¬ 
cess any cell of the expert system directory and by doing so retrieve informa¬ 
tion about the expert system to which it refers. The data shown in Figure 8.4 
represent one of the ways in which the information can be displayed to the 
user. It represents Albert’s expert system when Albert deals with the concept 
“Team Competence.” It tells us how important each attribute is to Albert, 
and it tells us the shape of the functions (positive, negative, concave, con¬ 
vex, or linear), helping us to understand more clearly how Albert uses attri- 
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Figure 8.5 

An Expert Systems Directory 


***** expert directory and status ***** 

-» * = * COMPLETED 1 P * ‘PATCHED FOR SCORING' 



CODE 


■ 





ALBERT LISTER 

ALBERTI 

10-26-1986 

F 


* 



* 

EVELYN ENDIVE 

EVELYN2 

10-26-1986 

* 


* 

* 



CHARLES LEWIS 

CHARLE3 

11-14-1986 


* 


* 

P 


ENRICO CALIFANO 

ENRICO4 

11-21-1986 


* 


* 


* 

LAURA CHAU 

LAURA#5 

11-21-1986 

★ 

* 


* 

* 


META-EXPERT 

META—6 

11-21-1986 

* 

P 

P 

P 

* 

P 

META-EXPERT 

META—7 

04-16-1991 

* 

* 

* 

* 

* 

* 


Concept Directory: 


A = Overall Merit 
B * Scientific Merit 
C = Team Competence 
D = Facilities Available 
E = Economic (Cost) Factors 
F = Public Health Impact 


Note: Asterisks indicate the existence of expert systems for the corresponding expert and con¬ 
cept. The letter P indicates that the expen system has been patched, so as to be used in 
the evaluation of alternatives. 


bute information when making judgments about proposals. Having this 
kind of information available is essential to users deciding how much weight 
to give to Albert’s expert system, or indeed, whether to include Albert at all. 

Ideally this information should be available for each concept of the hier¬ 
archy. Figure 8.6 displays indexes of relative importance for Charles Lewis 
for the concept “Scientific Merit.” Figure 8.6b is a graphic display of the 
weights shown in Figure 8.6a. 

Although this kind of information might well be considered when decid¬ 
ing how much weight to give to an expert’s evaluation of a concept, experts 
will often vigorously defend the importance they attach to attributes, and it 
is often better either to compromise these differences algebraically or to 
bring disparate experts together, asking them to resolve the differences. 

ALLOCATION AND INTEGRATION OF EXPERTISE: 

THE PROPERTIES OF EXPERTS 

Stylistic tendencies (Ramanaiah & Goldberg 1977) inherent in an individ¬ 
ual’s judgments provide a more cogent source of information that is ex¬ 
tremely useful in deciding how much weight to give to an expert and whether 
to patch a given expert to a concept. I refer here to the response styles and 
biases of experts, some of which have been described in the literature as 
common errors (Smith 1983) and to measures of the “worth” of the expert 
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Figure 8.6 

Display of the Relative Weights of Attributes 

Figure 6a. 

SELECTED CONCEPT = SCIENTIFIC MERIT 
CANDIDATE CLASS = PROPOSAL, PROPOSAL, NIH/NCI 
EXPERT: (C) CHARLES 
PROPOSAL.WTS 01-01—1992 09:59:01 

FIDELITY - 95.20 

RELATIVE WEIGHTS ANALYSIS 


COMPONENT WEIGHTS 


ATTRIBUTE 

LINEAR 

NON-LIN 

TOTAL 

SHAPE 

KNOWLEDGE OF LITERATURE 

9.37% 

1.83% 

11.20% 

POS . LINEAR 

SALIENCE OF PROBLEM 

20.37% 

-1.54% 

21.92% 

POS.LINEAR 

APPROPRIATENESS OF HYPOTHESES 

9.13% 

6.74% 

15.87% 

POS.CONCAVE 

DESIGN METHODOLOGY 

38.68% 

-3.94% 

42.62% 

POS . LINEAR 

ANALYTIC METHODOLOGY 

3.36% 

-5.04% 

8.40% 

NIL 

TOTAL 

80.91% 

19.09% 

100.00% 


VARIANCE EXPLAINED 

86.93% 

3.82% 

90.75% 



Figure 6b. 

SCIENTIFIC MERIT. EXPERT^ CHARLE3 PROPOSAL, NIH/NCI 

r DEDUCED RELATIVE IMPORTANCE WEIGHTS 

OF ATTRIBUTES INTUITIVELY USED BY EXPERT CHARLE3 (P) 

KNOWLEDGE OF LITE|i|iIl 11.2 % POS. LINEAR 

| SALIENCE OF PROBL^^^^|i|ii 21.9 % POS. LINEAR 

APPROPRIATENESS 15.9 % POS. CONCAVE 

;i DESIGN MSTHODOLCG ^^^^ i^^^^H^l 42.6 % POS. LINEAR 

| ANALYTIC METHODOXlllil 8.4 % NIL 

| O...10...20...30...40.. .50. . .60.. .70. . 

VARIANCE EXPLAINED 86.93% 3.82% 90.75% 


Note: The figure contrasts two formats for representation of the same data: tabular display 
(top panel) and graphics display (bottom panel). Both give insight into the relative 
weights of attributes intuitively used by an expen in the evaluation of “Scientific Merit.” 
The entries reflect some of the parameters estimated via a function-based expert system 
approach. 

system that is intended to reflect their judgments. Four such measures are 
considered here: 

1. Fidelity. The goodness of fit of the model to the expert. 

2. Standards. The extent to which the expert is “tough” versus “forgiving” when eval¬ 
uating alternatives. 

3. Discriminabiliiy. The extent to which the expert differentiates among alternatives, 
as opposed to using only a limited part of the scale. 
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4. Typicality. The extent to which the expert's viewpoint >cr ;udgn< 
cal of the viewpoints of other experts or stakeholders. 

Decision support software can be designed re display :hes« 
sures to the user and to implement algorithms for iggreg 
across experts in a manner that is responsive :e the kinds of 
and biases we have discussed. Function-based expert systems 
foility. 

Figure 8.7 displays tabular output of these indexes Wim 
overall merit of proposals (the firs: block of dan :r the lire 
(I) Evelyn is least typical, (21 all three expert systems ra% 
(3) Alben and Laura have lenient standards, whereas Eve.- 
pectations, and (4) discriminabilit} is high for ah three ex re 
The column to the light indicates the weigh: assignee m 
the expert, for purposes of creating a single model "0 rr 
group of stakeholders. Because the user has patched Vtxr* 
“Overall Merit ^ neither Evelyn’s nor Laura’s expertise s cm 
uating this concept. (Laura is also shown -as having zesr. ru 


Figure 8.7 

Indexes of an Expert’s Performsae* 
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model that evaluates concept D.) When more than one expert is to be in¬ 
cluded, an algorithm within the software implements a methodology capa¬ 
ble of aggregating the expert systems of up to twenty-five experts on the 
basis of the four factors shown. The algorithm generates parameter esti¬ 
mates for a single comprehensive model and computes the relative weight 
given to each expert included. This is done separately by concept. The 
weightings of experts are displayed in the right-hand column of Figure 8.7. 

That this kind of information is critical to decisions involving the weight¬ 
ing of experts should be self-evident. First and most obvious is the fact that 
unless the model of a given expert’s judgments is accurate, there is not much 
rationale for including the expert at all. Low fidelity scores may be indicative 
of a poor choice of models, but in practice they are more indicative of an 
expert who is uncertain or inconsistent in the use of information about the 
concept. Second, experts who are quite capable of ranking alternatives may 
not have a reasonable basis for setting organizational expectations in an ab¬ 
solute sense. Therefore, experts whose biases are such as to cause their judg¬ 
ments to reflect very low standards or expectations should be candidates for 
exclusion in organizations attempting to upgrade their standards. The con¬ 
verse is also true. 

Third, experts who experience difficulty in discriminating among alterna¬ 
tives may not be as effective as experts who discriminate well..Therefore, the 
discriminability scale provides a useful indicator for the differential weight¬ 
ing of an expert. Finally; experts whose weighting of attributes is extremely 
atypical of the majority of experts should be given a second look. It is possi¬ 
ble that the atypical expert knows something of which other experts are only 
dimly aware; in this case, a group discussion among experts concerning the 
parameters of their models can be enlightening. But it is more likely that 
the isolated expert is “off the wall,” in which case it would be unwise to give 
the identified expen system much weight. 

Function-based expert systems can deal with the fact that different experts 
or stakeholders will have different kinds of expertise and perhaps differing 
responsibilities with respect to the decisions being made in an evaluation 
study or in an organization or agency being evaluated. The foregoing discus¬ 
sion suggests that the problem of allocating and differentially weighting ex¬ 
pertise can be managed within an operating expert system environment. 

SENSITIVITY 

Sensitivity refers to change in the value of an expression that is associated 
with change in the value of one of its variables. Mathematically, sensitivity 
can be gauged by the value of the partial derivative, given a set of values (in 
this case attribute values) of the variables. Users of function-based expert 
systems can be guided to a clearer understanding of the orientations of their 
experts by observing the effects of these changes in a dynamic graphic envi¬ 
ronment. 
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An example of a sensitivity screen is shown in Figure 8.8. When the screen 
appears, users can adjust the values of the attributes, moving them into any 
pattern they wish. As they do so, they will see, on-screen, the effects of these 
changes on the concept being evaluated. It is akin to a what-if game. The 
question is of the form: If one changes the rating of, say, knowledge of liter¬ 
ature from a 7 to an 8, how much of a difference would that make to Charles 
Lewis’s evaluation of “Scientific Merit”? 

This kind of information might be of considerable use to users of expert 
systems as they attempt to decide how much weight to give to Charles 
Lewis’s expertise in evaluating the scientific merit of the proposals. It would 
be equally useful to an administrator attempting to decide whether to fund 
borderline proposals, given uncertainty in the information base. 

FLEXIBILITY 

Every evaluation directly or indirectly incorporates the personal utilities 
of responsible people within the organization. This being the case, it follows 
that when organizations change their priorities or when people change their 
viewpoints, the decision support systems of the organization should be able 
to adjust appropriately, so as to reflect these changes. 


Figure 8.8 
A Sensitivity Screen 


i i ■ « ■ SENSITIVITY ANALYSIS i 
EXPERT = CHARLE3 CHARLES LEVIS 


Partial derivative: 
dy/dx = -8.471 


CONCEPT SCORE =46.06 
MIN = 3.230 

MAX = 87.049 


USE CURSOR REVS TO CHANGE ATTRIRIITE U AMIES 

KNOWLEDGE OF LITERATI IRE 1. .2.3 . A .0 l...7. ft . .0 
SOL IENCE OF PROBLEM 1 .2. 0 A. .5 . .6..7. . 8 . .9 

APPROPRIATENESS OF HYPOTH . 2. 3.-4 ..5 ..6-.7. 0 .9 
DESIGN METHODOLOGY 1..Z.3..4..5..6.0..B..9 
ANALYTIC METHODOLOGY t. .2. 3 . Q .R. i>. .7. .8. .9 


: <—BELOW STANDARD: <MIN: <GOOD->: <-SUPER I0R: 
-A-A-A-A-A- 


SCIENTIFIC MERIT 


.10.. .20.. .38,. .40.. .58.. .68.. .70.. .80.. .98.. .99 


Note: As users move a cursor to adjust the values of the attributes, they observe the resulting 
change in the concept score. 
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For these reasons, the allocation and redistribution of expertise within an 
operating expert system become significant factors in its usability and in¬ 
strumental factors in decision support. The user may wish to add experts to 
an existing panel, delete others, or utilize an expert for evaluation of a dif¬ 
ferent concept. It is important to have the flexibility to do this with a few 
keystrokes rather than having to redo an entire system development process. 
Alternatively, an expert may develop new insights about a decision problem 
and wish to substitute a new model in place of his or her original one. It 
should be a simple matter for the software to effectuate the desired changes 
and to be able to reevaluate alternatives, based on a revised model. 

In the EXPERT87 environment, these changes are easily accomplished. 
The user can add or delete experts or change their degree of influence over 
the process with a few keystrokes. The properties of their expert systems can 
be reviewed on-screen, where decisions can be made about their inclusion or 
exclusion. And there are algorithms that assist the user in optimizing the 
allocation of experts to the various concepts of the decision task. Following 
these processes, the system generates and displays a table that indicates the 
precise manner in which expertise has been allocated by the system. One 
such table is shown in Figure 8.9. 

Figure 8.9 represents a culmination of the process of creating an expert 
system adapted to the user’s specific requirements. Upon completion of 
these steps, the user is ready to utilize the expert systems that have been cre¬ 
ated, for the purpose of evaluating alternatives. In this respect, one would 
wish to have available an expert system development tool that can also serve 
as a decision support system for the evaluation of alternatives and perhaps 
for guidance in making tough decisions. 


Figure 8.9 


Allocation of Expertise among a Committee 




«.=* a a— x. a rrjc 

XAOL - 





META-EXPERT = META- 

-6 



Entries 

are RWF's 

(relative weighting factors) 

for each 

Expert 


A 

B 

C 

D 

E 

F 

EXPERT 

OVERALL 

SCIENTIF 

TEAM 

FACIL. 

ECONOMIC PUBLIC HLTH 


MERIT 

MERIT 

COMPETNCE 

AVAIL. 

FACTORS 

IMPACT 

ALBERTI 

1.00 


0.58 



1.00 

EVELYN2 

0.00 



0.55 


0.00 

CHARLE3 


0.68 

0.42 

0.08 

1.00 


ENRIC04 




0.38 



LAURA#5 

0.00 

0.32 


0.00 

0.00 


TOTAL 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 


Note: The summary table indicates the contribution of each expen to the evaluation of pro¬ 
posals. The entries are calculated from the four indexes: fidelity, bias, discriminability 
and typicality; taking into account the user’s decision to include or exclude particular 
experts in the evaluation of particular concepts. 
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EVALUATION OF ALTERNATIVES: DECISION SUPPORT 

In function-based expert systems, the database for decisions is a matrix of 
attribute values. The rows of this matrix represent alternatives (programs, 
candidates, proposals, interventions, etc.), and the columns represent attrib¬ 
utes. The columns are grouped, so that sets of attributes that define con¬ 
cepts are clustered together. The process of evaluation is that of mapping the 
attribute values into objective measures of concepts and mapping concepts 
into an objective overall evaluation of each alternative. The expert systems 
created by the software serve as the transformation functions for this map¬ 
ping. 

It is of fundamental importance to realize that there is no single, unique 
formula for aggregating attribute values into measures of overall efficacy of 
alternatives. Evaluations are necessarily relative to the point of view of the 
stakeholders. This is as true in everyday life as it is in expert systems soft¬ 
ware. Therefore, decision support systems will be effective to the extent that 
they permit easy substitution of stakeholders and/or differential weighting 
of the points of view of stakeholders. One way of providing this mechanism 
was discussed in the previous section. 

Once the desired expert system is in place, the user may display evalua¬ 
tions in a variety of formats. In the EXPERT87 environment, these options 
are depicted in the menu screen shown in Figure 8.10. Selection of option C 
from this menu produces a graph, on-screen, of the son shown in Figure 
8.11a. Here, the seven proposals before the committee are ranked and evalu¬ 
ated with respect to the concept “Team Competence.” The lipoprotein pro¬ 
posal receives the highest rating, and the remote EKG monitoring proposal 

Figure 8.10 

A Menu Screen for the Evaluation of Alternatives 


Conceptual Dimensions 

1 Of MERIT OF PROPOSAL 

. .. ! 
Other functions / options 

A. MERIT OF PROPOSAL 

B. SCIENTIFIC MERIT 

C. TEAM COMPETENCE I 

D. FACILITIES | 

E. ECONOMY OF COST 1 

F. PUBLIC HEALTH IMPACT 

U. PROFILE. Evaluation of a 

I specific alternative. 

V. DISPLAY current Expert System. 

W. PRINT Evaluations of alternatives 

X. EVALUATION of an imported sen 

of alternative data. 

Y. PRINT Ranked alternatives frcr 

an imported set. f 

Z. RETURN to Main Menu 


Enter an option: 


FI * Esc 


ENTER * # 1 TO CHANGE ATTRIBUTES 



Note: The user may comparatively evaluate alternatives with respect to any selected concept. 
Alternatively, the user may select an alternative and review its evaluations on ai. con- 
cepts, as well as the overall evaluation. 
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Figure 8.11 

Evaluation of Two Concepts of Competing Proposals 

Figure 11a. 


* * * CONCEPTUAL LEVEL * * * 

CONCEPT - TEAM COMPETENCE EXPERT SYSTEM = META-EXPERT6 (P) 



Figure. 11b. 


* * * CONCEPTUAL LEVEL * * * 

CONCEPT - PUBLIC HEALTH IMPACT EXPERT SYSTEM = META-EXPERT6 (P) 



Note: The concepts team competence (top panel) and public health impact (bottom panel) are 
shown. Alternatives are automatically ranked by the program with respect to each con¬ 
cept. The vertical line indicates a score below which proposals fail to exceed the minimal 
standards of the committee. This cutting score is based on an algorithm that automati¬ 
cally captures and reflects each expert’s subjective standards. 

also qualifies for a superior rating. Two of the proposals fall below the mini¬ 
mum level. 

Selection of option F displays the same proposals, this time ranked with 
respect to “Public Health Impact” (Figure 8.11b). One can see by this exam¬ 
ple that the proposal named “Platelets and Type A” is ranked highest with 
respect to its potential public health impact, while all six of the remaining 
proposals are less than minimally acceptable. Since it was Evelyn’s expert 
system that was chosen for evaluation of public health impact, it might be 
useful for the committee to review both the characteristics of Evelyn (fidel¬ 
ity, standards, etc.) and the attribute ratings of the proposals, to achieve a 
better understanding of why the platelets and type A proposal received such 
a high rating on this concept, relative to the others. 
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Selection of option A causes all proposals to be ranked and displayed in 
terms of overall merit of proposal (Figure 8.12a). In the example, three pro¬ 
posals fall in the superior range, with the lipoprotein and the type A pro¬ 
posals being clearly superior. The survey of smoking fails to reach a minimal 
level of acceptability. 

Another form of decision support is provided by selecting option U. This 
enables the display of any individual alternative. In Figure 8.12b, I present 
the display that results from selection of the lipoprotein proposal. The pro¬ 
file provides a concise evaluation of the proposal. Team competence, facili¬ 
ties, and cost factors are evaluated as outstanding, scientific merit of the 


Figure 8.12 

Overall Rankings vs. Evaluation of Selected Proposals 

Figure 12a. 


CONCEPT * MERIT OF PROPOSAL EXPERT SYSTEM * ALBERTI (P) 




Note: Evaluation of the overall merit of all proposals is shown in the top panel. The bottom 
panel displays the evaluation of all concepts of a specific proposal selected by the user 
(lipoproteins & metabolism). 
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proposal is acceptable, although the public health impact is marginal. Ag¬ 
gregating over the five concept scores yields the overall evaluation of 80.8 
for this proposal. This is shown as the solid bar graph at the bottom of Fig¬ 
ure 8.12b. 

It is of interest that the overall rating of 80.8 is higher than the ratings of 
any of the five concepts on which it is based. Why should this be? To address 
this question, note first that the derived overall ratings are on an absolute 
scale, with a precise designation of the passing point. As with other parame¬ 
ters of the expert systems, the passing point is derived mathematically from 
information collected during the policy-capturing phase. It is therefore an 
accurate reflection of the standards of the experts selected by the user. From 
the heading at the top of Figure 8.12b we see that the overall evaluations are 
based exclusively on Albert’s expert system, although the evaluations of the 
component concepts shown in the figure are based on different combina¬ 
tions of experts. 

By referring to the first record of Figure 8.9, we see that the standards of 
Albert (28.0) are lenient, in contrast, say, to the standards of Evelyn (75.0). 
In this example, the user chose to override the software’s algorithm for the 
aggregation of the expertise of multiple stakeholders and chose instead to 
select Albert for the top-level evaluation. As a result, the overall evaluations 
of all proposals displayed In Figure 8.12a were high, and one result of this 
was an overall rating of 80.8 for the lipoprotein proposal. 

Had the user elected to permit the expert system aggregation algorithm to 
be invoked, the algorithm would have recognized the leniency inherent in 
Albert and Laura’s standards and would have given more weight to Evelyn’s 
point of view. That would have resulted in a downward shift in the cutting 
score by a constant amount. The relative rankings of the proposals would 
have remained unchanged, but all proposals would have fared more poorly. 

SUMMARY 

This chapter began with a distinction between rule-based and function- 
based expert systems. Some important limitations of rule-based expert sys¬ 
tems were noted. The main thrust of the chapter has been to clarify, by both 
discussion and example, the methodology underlying function-based expert 
systems and their application to program evaluation. 

Rule-based expert systems have, until now, been clearly dominant and 
have been used in many contexts with considerable success. However, there 
are many classes of problems for which these systems are ill suited and for 
which function-based expert systems can be employed to advantage. 

The versatility of function-based expert systems was noted, including 
their usefulness to Integrated decision support systems. These tools provide 
users with considerable flexibility. They assist in the evaluation of alterna¬ 
tives within the framework of standards and values of individual stakehold- 
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ers, and they are able automatically to aggregate the expertise of their 
stakeholders on a rational basis. 

The variety of feedback that becomes possible through adoption of func¬ 
tion-based expert systems can be presented with clarity in the form of screen 
displays. These displays can enable an organization to develop a more com¬ 
prehensive understanding of the process by which decisions are made among 
competing alternatives. At the same time, they offer stakeholders insights 
into the basis for their various points of view. It should also be noted that 
function-based evaluation systems convert a heretofore subjective process of 
decision making into one that is completely rational, while preserving, in 
their data files, the algebraic functions that form the basis of the decisions. 
This offers organizations a high degree of audit control over its decision¬ 
making functions, enabling an objective justification of decisions when re¬ 
quired. 

The most advantageous areas for application of function-based expen 
systems are in committee and organizational decision making. It is in this 
domain that it becomes vital both to understand the orientations and stan¬ 
dards of individual decision makers and to aggregate disparate orientations 
and standards in a cohesive and rational manner. Program evaluation, 
proposal evaluation, budgeting, and personnel decision making represent 
some of the more obvious challenges for application of function-based ex¬ 
pert systems. 

NOTE 

1. The examples presented in this chapter are fictitious. No implication is in¬ 
tended concerning the specific factors NIH committees might consider in their evalu¬ 
ations of proposals. 
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Knowledge-based Explanation in 
Multiattribute Decision Making 


Marko Bohanec and Vladislav Rajkovic 


In general, the decision-making problem can be defined as follows: 

Given a set of alternatives A = {a u a 2> . . .} and (somehow expressed) aims or goals 
of the decision maker, 

find alternative a { e A that best satisfies the goals. 

Problems of this kind can be found in almost any field of human activity, 
ranging from everyday personal decisions to more complex problems in 
economy, management, and medicine. The complexity of such problems 
usually originates in: 

• Complex and often incomplete, uncertain, or conflicting knowledge of how to de¬ 
fine and achieve the goals. 

• Numerous and/or loosely defined alternatives. 

• A large number of parameters that influence the decision. 

• The presence of several decision-making groups with different objectives. 

• Time constraints and other resource limitations imposed on the decision-making 
process. 

A number of methods and computer programs have been developed to sup¬ 
port decision makers in solving more or less complex decision problems 
(Humphreys & Wisudha 1987). They are usually studied within the frame¬ 
work of decision support systems (Keen & Scott Morton 1978; Alter 1980), 
operations research and management sciences, decision theory (French 
1986), or decision analysis (Phillips 1986). In a broad sense, expert systems, 
which originate in artificial intelligence, can also be considered systems that 
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solve and explain complex, repetitive decisions (Efstathiou & Mamdani 
1986; Turban 1988; Klein & Methlie 1990). 

In this chapter we show how multiattribute decision-making methods can 
be combined with expert systems-in particular how to support explanation 
and analysis of decisions. 


MULTL4TTRIBUTE DECISION MAKING 

In multiattribute decision making, the decision problem is decomposed 
into smaller, less complex subproblems (Keeney & Raiffa 1976; Chankong & 
Haimes 1983; French 1986). Such subproblems are represented by a set of 
criteria (or attributes) X = {xuXz* * * * » X n }- Criteria are variables that take 
values from the corresponding value domains X\>Xi> * • • Xn- 
Each alternative a e A is measured and described by a vector of values 

a = <v i# v 2 , . . . , v „>, v 2 e X,-, i = 1, 2, . . . , n. 

The alternative is then evaluated, usually in two steps. First, the decision 
maker’s preferences p i of a according to each separate criterion x, are ob¬ 
tained: 

Pi = ffy), i = 1 ? 2, . . . , n. 

Here, /■ is a partial utility function defined by the decision maker for each 
Xi- The preferences p i are usually expressed as numbers on a certain interval, 

for example [ 0 , 13 - 

The total utility of a is finally aggregated by the function F: 
u(a) = F(p„ p 2 , . . . , p n ). 

In practice, Fis most commonly expressed as a weighted sum where weights 

w. are defined by the decision maker: 


F\p.,p 2 , ... ,p„) = £ w,p,. 

i— 1 

The alternatives are finally ranked according to the total utilities. In general, 
the alternative with the highest utility should be selected as the best one. 

Note, however, that only the main ideas of multiattribute decision making 
have been presented; the details are treated quite differently in different meth¬ 
ods. For example, criteria may be additionally structured into trees. There are 
also considerable variations in the representation and assessment of alternatives 
and utility functions (French 1986; Humphreys & Wisudha 1987). 
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The most common drawback of existing multiattribute methods, at least 
for some classes of problems, is the need to translate the decision makers 5 
knowledge about a decision problem into numbers and functions. Experi¬ 
ence with expert systems (Turban 1988; Klein & Methlie 1990) indicates that 
this is not always necessary. There are decision problems in which qualitative 
judgment prevails over more or less exact quantitative evaluation. For such 
problems, it is quite a natural choice to use models that incorporate qualita¬ 
tive elements as well, for example, qualitative (descriptive, linguistic, ordi¬ 
nal) variables and rules. Expert system applications have confirmed that 
such an approach is not only feasible but can also provide some valuable 
features: explicit and flexible knowledge representation (Fox 1985), trans¬ 
parency (Levine, Pomerol & Saneh 1990), natural language dialogues (Vari 
& Vecsenyi 1988), and explanation of reasoning (Efstathiou & Mamdani 
1986). 

In the following sections we show how explanation facilities can be gained 
by combining multiattribute and expert system paradigms. The proposed 
approach is based on an explicit articulation of knowledge about a particu¬ 
lar decision-making problem. Knowledge representation is specific and ori¬ 
ented toward multiattribute problems. It consists of tree-structured criteria 
and utility functions that are represented by rules rather than formulas. 

KNOWLEDGE REPRESENTATION 

Knowledge representation for multiattribute decision making is based on 
a tree of criteria T = (X,S), where: 

X is a set of criteria {*i, Xi> - • - » X n }l each criterion may take values from the 
corresponding value domain X t . 

Sis a. mapping from X to the power set of X. For each criterion, S defines the set oi 
its immediate descendants (sons) in the tree. Only such mappings S are permu¬ 
ted that structure criteria into a single (connected) tree whose root is 

Criteria x for which S(x) = ® are leaves of the tree. They are also referred 
to as basic criteria. The remaining ones are called aggregate criteria. 

Domains X l9 X 2i . . . , X n are assumed to be discrete and finite. They 
generally consist of words, not numbers; words such as good or low can be 
used. Numerical qualities can be expressed as intervals of values; each inter¬ 
val is treated as a discrete domain element. It is also recommended to order 
domain elements by preference from “bad” (nonpreferred) to "'good (highh 
preferred) values, for example {bad, acceptable, good, excellent] or, for 
price, {high, medium, low}. 

The tree of criteria represents the structure of a particular decision prob¬ 
lem. The purpose of the second component of the decision know ledge base, 
utility functions, is to define the relation between aggregate criteria and their 
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descendants in the tree. For each aggregate criterion Xk> the corresponding 

utility function 

F k : X kl x X kl x . . . X km X k 

should be defined by the decision maker. Here, X k{ , ...» X km denote the 
value domains of ail the descendants of Xk- 
When different decision groups with different interests are involved in the 
decision-making process, each group may define its own set of utility func¬ 
tions. Therefore, more than one function can be defined for each aggregate 
criterion, as shown in Figure 9.1. 

Utility functions F k are represented by elementary decision rules of the 

form 

if = v *i and • • * and x kfn = then Xk = v*, 

where v k; and v k denote single values taken from domains X k . and X k9 respec¬ 
tively. 

The consistency of rules is required. This means that the rules should be 
unique with respect to their conditional parts. On the other hand, the com- 


Flgnre 9.1 

Criteria Tree with Utility Functions and Alternatives 
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pleteness of rules is not required, meaning that for some combinations of 
values v kl , . . . , v kfn there might be no corresponding rules. The set of rules 
that correspond to a particular aggregate criterion is usually represented in a 
tabular form (see, for example, Table 9.2). 

After the criteria tree and utility functions have been defined, the evalua¬ 
tion of alternatives can start. The alternatives are first measured and de¬ 
scribed by values of basic criteria. The utility functions are then applied in a 
bottom-up manner in order to obtain aggregate values for each alternative. 
These values, particularly the one that has been assigned to the root, are 
finally used to select the best alternative. 

Figure 9.1 illustrates a decision knowledge base after alternatives have 
been evaluated. There are two aggregate criteria, Xi (the root) and * 4 . The 
corresponding utility functions are F x and F 4 . Note that in the presence of 
different decision-making groups, more than one function can be defined 
for each aggregate criterion. 

There are two alternatives shown in Figure 9.1. They are described by val¬ 
ues v 2 , v 3 , v 5 , and v 6 that are assigned to basic criteria * 2 , * 3 , * 5 , and Xe- 
Evaluation results v 4 and v x for each alternative have been obtained by func¬ 
tions F 4 and F ls respectively. The evaluation is performed separately for each 
decision-making group. Therefore, when there are more groups, a set of 
evaluation results, consisting of v 4 and Vj, is obtained for each group. 

An example of a real-world criteria tree is shown in Figure 9.2. The tree 
was designed in a project whose goal was to evaluate microcomputers with 
respect to their appropriateness for education in primary and secondary 
schools of the Republic of Slovenia (Rajkovic et al. 1985). Figure 9.2 actu¬ 
ally presents only the subtree that was used to evaluate microcomputer hard¬ 
ware. The whole tree, which additionally included software and commercial 
criteria, was about twice as big. In Figure 9.2, aggregate and basic attributes 
are represented by uppercase and lowercase letters, respectively and fol¬ 
lowed by the corresponding value domains. According to the tree, micro¬ 
computer Hardware is determined by three main groups of criteria: 
Environment, Processor, and capabilities to Upgrade the system. The first 
two are additionally decomposed. For example, the Environment Is decom¬ 
posed into Basic Communication, Peripherals, and Connections. 

Table 9.1 presents a simple set of elementary rules that define Basic Com¬ 
munications according to the quality of Monitor and Keyboard. Note the 
incompleteness of this table: some combinations of values of Monitor and 
Keyboard, for example Monitor = good and Keyboard = good, are undefined 
in the table. 

The approach presented in this section has been implemented within an 
expert system shell DECMAK (Bohanec, Bratko & Rajkovic 1983; Bohanec 
& Rajkovic 1987; Rajkovic, Bohanec & Batagelj 1988) and thoroughly 
verified in practice. The shell interactively supports the decision maker in 
the knowledge acquisition and evaluation stages of the decision-making 
process. 
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Figure 9.2 

Criteria Ttee for the Evaluation of Microcomputer Hardware 


t HARDWARE 

- ENVIRONMENT 

-# BAS.COMM 

-• monitor 

-• keyboard 

-♦ PERIPH 

-• printer 

-1 EXTJMEM 

-• cassettes 

-• disk 

-• add.periph 

-* CONNECTIONS 

I-• loc .network 

INTERFACE 

-• std Jnterf 

—• a/d.conv 

|-• upgrade 

1 -1 PROCESS 

-• GRAPHICS 

-• bw .graph 

-• col .graph 

-• characters 

-• memory 


unaccept, accept, good, exc 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
no, yes 

none, accept, good 
no, yes 

unaccept, accept, good 
none, accept, good 
unaccept, accept 
no, yes 
no, yes 

bad, accept, good 
unaccept, accept, good 
unaccept, accept, good 
unaccept, accept, good 
none, accept, good 
unaccept, accept, good 
unaccept, accept, good 


It is beyond the scope of this chapter to describe the knowledge acquisi¬ 
tion techniques that are used in DECMAK. They are based on an interactive 
dialogue for the acquisition of elementary decision rules. Additional tools 
are available that present the whole knowledge base to the decision maker 
using different viewpoints and levels of detail (Rajkovic, Bohanec & Batagelj 
1988; Bohanec and Rajkovic 1987, 1988). 

The following sections are focused on techniques and algorithms used in 
DECMAK to support the evaluation stage of the decision process. In this 


Table 9.1 

Elementary Decision Rules for Basic Communications 


MONITOR KEYBOARD 

BAS.COMM 

unaccepi unaccept 

unaccept good 

accept accept 

accept good 

§ood unaccepi 

good accept 

unaccept 

unaccept 

accept 

accept 

unaccept 

good 
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stage, the evaluation of alternatives, explanation of results, and analysis of 
alternatives take place. 


EVALUATION OF ALTERNATIVES AND EXPLANATION 
OF RESULTS 

In this stage of the decision-making process, the knowledge base, which 
we assume has been defined and verified in the previous stages, is used to 
evaluate alternatives. At the beginning, the decision maker describes each 
alternative a e A by a vector of values that correspond to all basic attributes 

Xbi* Xhj' * * * » Xb m ' 


a = < V v 6 ,, .... v b J, v b . e X v / = 1, 2, . . . , m. 

In DECMAK, these descriptions are entered interactively, using a spread¬ 
sheet-like editor of alternatives. 

DECMAK is also capable of handling incomplete and uncertain data 
about alternatives by means of probabilistic and fuzzy distributions of val¬ 
ues (Bohanec, Bratko & Rajkovic 1983). For simplicity, this feature is omit¬ 
ted from the following presentation. 

After the alternatives have been described, they are evaluated by DEC¬ 
MAK. The evaluation proceeds from the bottom to the top of the criteria 
tree. The value v ; - of each aggregate criterion Xi is obtained by evaluating the 
corresponding utility function F t . The function arguments are the values v f . 
that are assigned to all the descendants of Xi in the tree: 

Vi = v, 2 , . . .), V tj G X,.. 

In order to evaluate F h DECMAK searches for the decision rule that exactly 
matches the arguments. If such a rule exists, it directly assigns the value to 
V/. Otherwise, the regression analysis is applied over some of the remaining 
rules. Only the rules that slightly differ from the arguments of the function 
are considered. The Euclidean distance is used as a heuristic measure of dif¬ 
ference between rules. 

The evaluation of alternatives assigns values (utilities) to all the aggregate 
criteria in the tree. These values determine the adequacy of each alternative 
according to the decision maker’s preferences. The alternative that has the 
most desirable values should be chosen. 

Some important questions usually emerge at this point: 

♦ How were the results obtained? Why they are such? Are they appropriate? 

• Are the results influenced by changes of basic criteria values, and how? Which 
changes are required in order to get a better (or worse) alternative? 
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• What are the advantages and disadvantages of a particular alternative? How does 

it compare to some other alternative? 

The evaluation results themselves are insufficient to answer these questions. 
Additional tools that help the user to explain, justify, and analyze the deci¬ 
sion are needed. As shown below, the rule-based approach offers quite use¬ 
ful support for such tasks. 

Obviously, the evaluation process itself can be easily explained in terms of 
decision rules that have been triggered and criteria values that have been 
used. This is similar to the How? type of explanation in expert systems. 

There is, however, a problem with the explanation of results obtained by 
the regression analysis. Due to its inherent “black box” functioning, it is dif¬ 
ficult to explain the computation itself. However, since it operates on a lim¬ 
ited set of rules (usually, two to five), these rules are shown to the user. 
According to our experience, the users are able to justify the appropriateness 
of the computation on the basis of this implicit information. 

With DECMAK, the evaluation and explanation are performed interac¬ 
tively. The user can focus attention on particular aggregate criteria, inspect 
and/or change rules, and follow the detailed explanation. This is often com¬ 
bined with what-if analysis, where the user changes descriptions of particu¬ 
lar alternatives, reevaluates them, and compares the results with the original 
ones. 

SELECTIVE EXPLANATION AND COMPARISON 
OF ALTERNATIVES 

The explanation described above answers the question: How were the 
results obtained? Its main role is to present a detailed overview of the evalua¬ 
tion process and to help the user in discovering errors in the knowledge base 
and descriptions of alternatives. However, there is a question that is usually 
more important in practice: Why are the results such? Unfortunately, the 
ultimate answer is hidden in rules; they are basic facts that can be meaning¬ 
fully explained only by the person who defined them. 

Nevertheless, some implicit and practically important answers can be 
gained by exploring the evaluation results and relations among them. For 
example, if an alternative got a bad final utility, one might be interested in 
sources (i.e., basic criteria values) that caused this result. For a better over¬ 
view of an alternative, it is also useful to find its most important advantages. 

For this purpose, a selective explanation algorithm is used in DECMAK. 
Its main role is to explain a particular alternative in terms of its most advan¬ 
tageous or disadvantageous characteristics. The explanation is selective be¬ 
cause, instead of all, usually quite numerous criteria, only the most relevant 
ones are shown in the form of subtrees of the whole criteria tree. 

This algorithm operates by searching for such maximal connected sub- 
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trees of the whole criteria tree in which a particular alternative evaluates to 
very bad or, alternatively, very good. To determine whether a value is very 
bad or very good, a simple heuristics is used. Since the criteria domains are 
usually preferentially ordered (95 percent of domains used in about forty 
practical applications of DECMAK were such), the leftmost element of each 
domain is considered very bad and the rightmost very good by default. 
When this is inappropriate, the user is allowed to redefine explicitly the sets 
of very good and very bad values. 

An example of selective explanation is shown in Figure 9.3. The quality of 
hardware of a particular microcomputer is explained on the basis of the cri¬ 
teria tree from Figure 9.2. The default settings have been used in order to 
determine very bad and very good values. 

Virtually the same algorithm can be used also for the selective comparison 
of alternatives, which is needed for comparison of two different alterna¬ 
tives, discovering changes of one alternative during the what-if analysis, and 
comparison of the same alternative that was evaluated by utility functions of 
different decision-making groups. 

Figure 9.3 

Selective Explanation of a Microcomputer’s Hardware 
Advantages: 

COL.GRAPH is good 

INTERFACE is accept 

because STD.INTERF is yes 
because A/D_CONV is yes 

ADDJPERIPH is yes 

CASSETTES is yes 

Disadvantages: 

HARDWARE is unaccept 

because ENVIRONMENT is unaccept 

because BAS-COMM is unaccept 

because MONITOR is unaccept 
because CONNECTIONS is unaccept 

because LOC-NETWORK is none 
because PROCESS is unacc 

because GRAPHICS is unaccept 

because BW.GRAPH is unaccept 
because CHARACTERS is bad 
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The comparison algorithm differs from the explanation one only in the 
condition that must be satisfied in all nodes of the extracted subtrees. Here, 
the condition is satisfied if the first alternative is (strictly) better than the 
second one or, alternatively, the second one is better than the first. The rela¬ 
tion better (represented by > in Figure 9.4) is again determined from the 
preferentially ordered domains or explicitly given by the user. Figure 9.4 
presents an example of selective comparison of two microcomputers. 

OPTION GENERATION 

Another important aspect of the analysis and implementation of the deci¬ 
sion is to find such small changes in descriptions of an alternative that cause 
degrading or upgrading its performance. The findings about changes that 
might degrade the alternative can be particularly useful in the stage of its 
implementation; once the most critical characteristics have been identified, 
it is much easier for the implementer to treat them carefully. On the other 
hand, the information about the changes that upgrade the alternative can 
serve as a guideline for possible improvements of the alternative. 

In DECMAK, such information can be gathered by an option generator. 
To describe the algorithm, let us concentrate on a single aggregate criterion 
y. Let Xi, Xii * • • > Xk be its descendants in the criteria tree. Therefore, the 
relation between the criteria is that 

y = Xi . x k )- 


Figure 9.4 

Selective Comparison of Two Microcomputers 

Alternative A is better in: 

BAS.COMM is better (good > accept) 

because MONITOR is BETTER ( good > accept) 

ADD_PERIPH is better ( yes > no) 

Alternative B is better in: 

HARDWARE is better (exc > accept ) 

because PROCESS is better ( good > accept) 

because GRAPHICS is better (good > accept) 

because BW_GRAPH is better (good > accept) 
because COL.GRAPH is better (good > none) 

PRINTER is better (good > accept) 
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We also assume that an alternative a e A has been already evaluated on y 
and denote with 

v = <v 1? v 2 , . . . , v*> 

the vector of values that have been assigned to Xu AT 2 » • • * * Xk for this alter¬ 
native. Let u be the utility of the alternative, obtained by 


u = F(v). 


The task of the option generator is then to determine the smallest changes 
Av of vector v which would lead to 

F(v + Av) = z , 

such that z > u for upgrading the alternative or z < u otherwise. 

At the beginning, the option generator implicitly completes the table of 
rules that define F by adding all the missing rules; the regression analysis 
from the previous section is used for this purpose. Then, the set of all rules 
that yield the value of z is determined. These are the rules of the form: 

if X\ = W\ and . . . and Xk = v k then y = z. 

Each such rule corresponds to the assignment of values 

Z = F(w u . . . , w k ) = F(w). 

Therefore, F(w) = F(v 4- Av), so 

Av = w — v. 

However, such Av need not be the smallest one; it should be compared to all 
the difference vectors that are determined from the remaining rules that as¬ 
sign y = z. Therefore, all pairs of rules such that z = F(w } ) and z = FO 2 ) 
are compared by the option generator. When upgrading alternatives, Avj 
that follows from the first rule is eliminated if w u > w 2J for all / = 1,2, 
When finding the smallest changes that degrade alternatives, the 
above condition is replaced by w ui < w 2 
The extension of this algorithm from one aggregate criterion y to any sub¬ 
tree of y is straightforward; the algorithm should only be applied recursively 
on all the aggregate descendants of y. An example of the option generator 
output is shown in Figure 9.5. It was generated by applying the above al¬ 
gorithm on rules from Table 9.1. Note, however, that this table is extremely 
simple, so the output is not as interesting and valuable as when treating 
more complex tables and subtrees. 
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Figure 9.5 

Output of the Option Generator 

Value 1: MONITOR— accept 
Value 2: KEYBOARD ==$<W 
Utility: BAS-COMM =accept 


BAS-COMM becomes unaccepi 

if ( MONITOR becomes unaccept ) or 
if ( MONITOR becomes good and 
KEYBOARD becomes unaccept ). 


BAS-COMM becomes good 

if ( MONITOR becomes good and 
KEYBOARD becomes accept ). 


APPLICATIONS 

The DECMAK system has been applied in about forty practical decision¬ 
making problems. They varied from simple, personal decisions, like job or 
car selection, to complex group decision problems, such as: 

1. Evaluating mainframe computer systems (eight applications so far). 

2. Selection of computers for schools (four applications). 

3. Microcomputer selection. 

4. Evaluation of production control software. 

5. Database management system selection (two applications). 

6. Trading partner selection. 

7. Evaluation of applications for nursery schools. 

8. Matching people to jobs. 

9. Expert team selection. 

10. Project feasibility estimation. 

11. Performance evaluation of public enterprises. 

The first part of the list (applications 1 to 5) indicates that DECMAK has 
been extensively used in decisions related to computers. The main reason for 
this is the computer science background of the authors of the system. In 
these problems, it was convenient for the authors to play a dual role of deci¬ 
sion analysts and computer consultants. Nevertheless, the applications num¬ 
bered 6 to 11 show that the usefulness of DECMAK is not limited to 
computer-oriented problems. The complexity of a particular decision prob¬ 
lem is reflected in the size of the knowledge base and the number of consid¬ 
ered alternatives. These data for some of the above problems are presented 
in Table 9.2. 
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Table 9.2 

The Complexity of Some DECMAK Applications 


Application 

Number of 

Basic 

criteria 

Aggregate 

criteria 

Alterna¬ 

tives 

La 

20 

9 

6 

b 

67 

45 

2 

c 

17 

9 

3 

a 

35 

24 

S 

2.a 

11 

6 

6 

b 

12 

9 

10 

c 

36 

19 

24 

5. 

74 

29 

3 

6. 

22 

12 

>20 ! 

7. 

8 

5 

>2000 

9. 

16 

S 

164 

10. 

34 

19 

1 

11. 

12 

9 

55 


DISCUSSION 

One of the main motivations for combining multiattribute decision mak¬ 
ing with the knowledge-based approach in DECMAK was to improve the 
quality of decision making by the aid of explanation. There are extremely 
rare decision situations where the criteria and alternatives are so dear that 
they directly lead to optimal solutions without any further analysis and ex¬ 
planation. Usually, a great deal of uncertainty and missing information is 
involved, important criteria may be overlooked or treated inappropriately, 
and so on. Therefore, the obtained results should not be accepted until they 7 
are carefully investigated, justified, and understood by the decision maker. 

The proposed approach is based on a structured and explicitly articulated 
knowledge base that can be argued, verified, and documented. Its structure 
is especially adapted for multiattribute decision problems. Although the 
structure is relatively simple compared to common expert systems, it has 
been found quite general and flexible in practice. In the stage of knowledge 
acquisition, the main advantages are easy acquisition of elementary 7 decision 
rules by an interactive human-machine dialogue and powerful capabilities 
for explaining the whole knowledge base by means of machine learning 
techniques. These are presented and discussed elsewhere (Bohanec & Raj- 
kovic 1988; Rajkovie & Bohanec 1991). 

In the evaluation stage of the decision-making process, the knowledge 
base is applied consistently to all alternatives in order to obtain and explain 
their utilities. The evaluation approximately corresponds to reasoning in 
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common expert systems. However, due to the specifics of the problem, it is 
much more function oriented; it basically calculates function values. There 
is no inference based on logic, theorem proving, or anything else. The order 
of evaluation of rules is strictly determined by the structure of criteria. 

In all practical applications of DECMAK, some kind of explanation was 
needed (and done) in the evaluation stage. Usually at least some of the eval¬ 
uation results were in disagreement with the decision maker’s expectations. 
It was necessary to investigate them in more detail and find out the reasons 
for the disagreement, which could have been caused by erroneous rules, in¬ 
appropriate descriptions of alternatives, overlooked criteria, or wrong ex¬ 
pectations. In this respect, the explanation by tracing the triggered rules has 
been found useful but not as much as we had originally expected. It was 
invoked quite rarely, mainly to explain unexpected values assigned to a single 
aggregate criterion. In our opinion, the main reason for this is in low and 
nonselective informativity of this kind of explanation. The decision maker is 
usually at the same time the author and user of the rules. For this reason, the 
triggered rules hardly provide any new information for him or her. 

In order to discuss alternatives within the decision-making group and to 
justify the decision, an overall (aggregate) explanation of each alternative is 
needed. In practice, it seems that the selective explanation algorithm suits 
extremely well for this purpose. It was used in all practical decision prob¬ 
lems. The decision makers considered it an important support tool for the 
explanation and verification of the decision. Its results were also very useful 
in preparing reports and other presentations related to the decision. Usually, 
only the translation of the printouts (such as the ones in Figure 9.3) into 
ordinary 7 sentences was sufficient to provide a comprehensible explanation. 

The main power of the selective explanation is in its ability to extract and 
group together the criteria and values that most relevantly determine the 
quality of a particular alternative. However, in the current implementation 
of DECMAJC, the “relevance” of criteria is determined by a simple heuristic 
inspection of criteria domains. Further research is suggested at this point to 
find more general and robust measures of relevance, for example, by taking 
into account the underlying utility functions. 

The option generator is usually applied after a limited set of possible can¬ 
didates for the best alternative has been identified. These are then thor¬ 
oughly analyzed, mainly to isolate critical characteristics that can easily 
degrade them in the implementation stage. When the decision makers are in 
a position to influence the alternatives, for example, to require some modifi¬ 
cations of computer hardware from the distributor, the possibilities of up¬ 
grading them are also interesting for investigation. According to our 
experience, the usefulness of the option generator heavily depends on the 
decision problem. There were problems where it was not needed at all, for 
example, in selecting an expert team and evaluating application for nursery 
schools. Its role was more important in decisions related to computers and 
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public enterprises. However, there was a problem when a decision had to be 
made as to whether to continue or abandon a large software development 
project. All aspects of this project were thoroughly analyzed almost exclu- 
sively by the option generator. 

CONCLUSION 

The decision-making approach presented in this chapter combines two 
techniques, multiattribute decision methods and expert systems. It is based 
on an explicit articulation of knowledge about a particular decision prob¬ 
lem. The knowledge is represented by tree-structured criteria and utility 
functions, defined by elementary decision rules. 

The approach is supported by an expert system shell, DECMAK, consist¬ 
ing of tools that support interactive knowledge acquisition and evaluation of 
alternatives. The main emphasis is on the explanation and analysis of the 
evaluation process. 

DECMAK has been verified in about forty practical decision-making situ¬ 
ations. The results confirmed the importance of the explanation facilities. 
Their availability motivated the decision makers to think about, verify, jus¬ 
tify, and explore decisions. We strongly believe this is one of the key elements 
that can improve the effectiveness and justification of decisions. 

Among the algorithms presented in this chapter, the selective explanation 
one has been found the most valuable in practice. Its main advantage is the 
ability to explain alternatives by selecting only the most relevant information 
and presenting it to the user in a structured form. 
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Decision-Aiding Software for 
Group Decision Making 


Floyd Lewis 


COLLABORATIVE WORK SUPPORT 

Modern organizations increasingly are using groups to make decisions 
and carry out tasks. Researchers have estimated that top management may 
spend up to 80 percent of their time in meetings (Mintzberg 1971; Van de 
Ven 1973), clearly a major commitment of resources for a typical company. 
Most expect that this trend will continue for the foreseeable future. 

In the past few years, there has been a notable increase in the attention 
given to the computer support of work that involves groups. Researchers 
have addressed many aspects of group work, such as project scheduling, 
document preparation, communications, and decision-making meetings. A 
variety of computer-based systems now exist to support many of these activ¬ 
ities. (For a collection of papers on collaborative work support, see Greif 
1988.) This chapter will focus on a specific system, MeetingWare, designed 
to support small groups engaged in face-to-face decision-making meetings. 


Problems in Group Decision Making 

Groups have encountered many well-documented difficulties in trying to 
work together to solve problems and make decisions: an overemphasis on 
social-emotional rather than task activities (Delbecq, Van de Ven & Gustaf¬ 
son 1975), failure to define a problem adequately before rushing to judg¬ 
ment (Maier & Hoffman 1960), pressures for conformity that can reduce 
creativity and result in “groupthink” (Chung & Ferris 1971; Janis 1981), and 
deindividuation and diffusion of responsibility that may lead to risky deci¬ 
sions (Diener 1980; Latane, Williams & Harkins 1979). 
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Group Decision Support Systems 

Over the years, researchers have developed a variety of techniques to try 
to overcome these difficulties. Brainstorming (Osborn 1963) and nominal 
group technique (Delbecq, Van de Ven & Gustafson 1975) are familiar exam¬ 
ples of typical structured approaches to group decision making. Recent at¬ 
tempts to integrate some of these techniques with computer technologies 
have resulted in the creation of a new set of computer-based tools, group 
decision support systems (GDSS). Huber (1984) has defined a GDSS as 
“software, hardware, and language components and procedures that sup¬ 
port a group of people engaged in a decision-related meeting.” A representa¬ 
tive sample of recent GDSS research includes papers by Bostrom (1991), 
Dennis et al. (1988), Dennis, Nunamaker, and Vogel (1990-1991), Easton et 
al. (1990), Ellis, Rein, and Jarvenpaa (1989-1990), Grohowski et al. (1990), 
Jessup, Connolly, and Galegher (1990), Keleman, Garcia, and Lewis (1989), 
Lewis and Keleman (1990), Lewis, Keleman, and Garcia (1990), McGoff et 
al. (1991), Nunamaker et al. (1989), Sanders, Cerveny, and Wang (1991), 
Tan, Wei, and Raman (1991), Venkatesh and Wynne (1991), and Vogel et al. 
(1989-1990). 

DeSanctis and Gallupe (1985) have identified four types of GDSS based 
on proximity (close versus dispersed) and duration of the session (limited 
versus ongoing). They 7 call a GDSS for meetings of limited duration where 
members are in close proximity (face-to-face) a decision room approach. If 
the participants are still relatively close together (in the same building) but 
the session is ongoing, they call it a local decision network. A GDSS de¬ 
signed for meetings of limited duration where participants are dispersed is 
classified as teleconferencing. Finally, GDSS for dispersed meetings of an 
ongoing nature are termed remote decision making . 

Another characteristic that may vary between GDSS technologies is the 
size of the group they are designed to support. Most seem to focus on groups 
of fewer than ten members, but some have been used with larger groups, 
such as the planning groups using the University of Arizona’s MIS Planning 
and Decision Laboratory, which were reported to exceed twenty members 
(Nunamaker, Applegate & Konsynski 1987). 

The GDSS described in this chapter, Meeting Ware, would be classified as 
a decision room approach by DeSanctis and Gallupe (1985). We designed 
MeetingWare for face-to-face interaction in sessions of limited duration, 
though several sessions may be needed to reach a decision. It is also a system 
best suited for small group use, where there are ten or fewer participants in 
any one meeting (although up to twenty participants are allowed by the soft¬ 
ware). 

Only a few GDSS systems have been developed to support face-to-face 
meetings. While most of these have been primarily used in laboratory set¬ 
tings, a few' early field studies are now available (for example, see Dennis, 
Nunamaker & Vogel 1990-1991; Lewis & Keleman 1990; and Nunamaker et 
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al. 1989). While Meeting Ware does have some unique features, for the most 
part it is representative of the approaches used by the majority of GDSS 
developers at this time. 

SYSTEM DESCRIPTION 

The Meeting Ware GDSS is typically installed in a decision room especially 
arranged for small group meetings (Figure 10.1). A typical decision room 
has five to ten microcomputers arranged on tables in a U or horseshoe 
shape, connected by a LAN (local area network). Laptop personal com¬ 
puters (PCs) are ideal for this application; their small size makes them unob¬ 
trusive in a meeting and nonthreatening to novices. In addition, it makes the 
system portable. Sessions can be moved to a client’s location or a conference 
facility or retreat. 

The group participants generally directly use all but one of the microcom¬ 
puters as personal workstations. A more powerful micro provides the net- 


Figure 10.1 

MeetingWare Decision Room 
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work shared disk and is used by the meeting “chauffeur” (a person trained in 
using the MeetingWare system). This system collects the inputs from indi¬ 
vidual participants and combines them into a group product. In addition to 
the chauffeur, there may be a group “facilitator” who may have worked with 
the group to design the script and who may remain with the group to help 
them with discussions and process questions. Typically, the facilitator does 
not use a micro during the meeting unless he or she is filling the role of 
chauffeur as well. 

While all the PCs are connected as a LAN, the chauffeur station is also 
connected to a display device like a video projector or an LCD plate for use 
with overhead projectors. This allows information to be displayed to the 
group on a screen or white wall at the front of the room. When using some 
MeetingWare tools, a second PC is connected to another display device, so 
that two displays can be shown simultaneously. A printer is connected to the 
chauffeur station to provide hard-copy records of the group’s activities as 
they proceed through the script. Typical output would be lists of ideas, rat¬ 
ings, graphs, and summary charts and tables. Participants no longer have to 
wait several days until someone types up the minutes to have a written record 
of their decisions. 

MeetingWare has been under development for ten years. It has evolved 
from an earlier GDSS called FACILITATOR, built at the University of 
Louisville in the early 1980s. MeetingWare now consists of an expandable 
set of modules that users can flexibly combine in a variety of ways to meet 
the needs of a specific group with a particular problem or decision. The 
process of selecting modules and integrating them into a particular sequence 
we call “script writing.” With the support of a MeetingWare tool module, 
script writing allows a meeting manager to specify all the GDSS tools that 
the group will use in a meeting ahead of time, along with the tool content 
and the sequence in which they will use the tools. This script is saved as a 
file; during an actual meeting session, the script is simply executed in order 
to call up the tools in the proper order. This process is shown in Figure 10.2. 

The current set of MeetingWare modules can be divided into six major 
categories: idea generation, Idea discussion, idea organization, idea evalua¬ 
tion, information sharing, and session management. The Symphony soft¬ 
ware package from Lotus and the QuickBasic language from Microsoft were 
used for generating the GDSS modules. 


List Generation 

There are two modules that a group can use for the purpose of generating 
a list of ideas: LIST and BRAINSTORM. The LIST module allows each 
participant to enter up to three ideas on a private screen. Participants work 
independently and do not see anyone else’s ideas during idea generation. 
The individual lists are combined and saved in a master file that can then be 
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Figure 10.2 

Script Writing in MeetingWare 

PROBLEM CONTEXT 



used by other modules (for discussion, editing, or organization). This mod¬ 
ule is used when independent thought is a desired feature. 

The BRAINSTORM module takes a different approach. Participants ini¬ 
tially enter ideas on their own screens, but these ideas are immediately sent 
to a common file and displayed on the main screen at the front of the room 
where everyone can read them. The combined file can serve as input to other 
modules, in the same fashion as the LIST file. This approach is generally 
used when the group wants to piggyback off each other’s ideas. 

It is also possible to use a modified approach with BRAINSTORM, when 
the front screen is not turned on until everyone has entered one or more 
ideas. This approach has the advantage of initial independent idea genera¬ 
tion while retaining the possibility for piggybacking later in the process. 
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Typical LIST and BRAINSTORM screens are shown in Figures 10.3 
and 10.4. 


List Discussion 

There are two modules in this category: DISCUSS and COMMENT. Re¬ 
search on group decision making has shown that ideas at the top of a list 
often receive much more discussion time than ideas on the bottom (Delbecq 
et al. 1975). This is closely related to the “valence of solutions” concept as 


Figure 10.3 

MeetingWare List Module 


| rteet.ingUa.re ITEM INPUT 



Figure 10.4 

MeetingWare Brainstorm Module 


! "heetingllarg BKftINSTQHH | 


** Please type your itens on the line belou. 

* Press the <ENTEH> hey after each item. 

* Uhen coMpletely done, press the CF183 key. 


liens 


EF13 = HELP 


IF1B1 = QUIT 
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discussed by Maier (1967) and Hoffman (1982). DISCUSS is designed to 
even out the amount of attention paid to ideas on the list so that all ideas get 
a fair hearing. 

DISCUSS reads a list of items from an ASCII text file and then manages a 
timed discussion of the items using the main shared screen at the front of the 
room. The group sets the amount of time to consider each item at the begin¬ 
ning of the process (we recommend two minutes per item initially). 

The group will then see the DISCUSS screen where the items are dis¬ 
played, one at a time, along with a timer and some instructions (Figure 
10.5). The group is instructed to determine whether each item is appropriate 
(belongs on the list), unique (has not already appeared on the list), and sin¬ 
gular (does not contain multiple ideas). After considering these criteria, the 
chauffeur may enter an idea into a structured outline in the ORGANIZE 
module, which is often executed simultaneously with DISCUSS. After the 
items are inserted into the outline, the group may discuss the merits of the 
ideas. The chauffeur can interrupt the timer at any time to move on to a new 
idea, or the group can extend the discussion beyond the allotted time if they 
desire a more thorough analysis. 

COMMENT reads a list of topics from a file and then opens a modified 
BRAINSTORM session, where each topic is displayed one at a time at the 
top of the shared screen at the front of the room (Figure 10.6). Participants 
can then type in their comments about the topic, and each comment will 
appear on a list on the front shared screen. The author remains anonymous. 
Participants can assign codes, which the software can use to group the com¬ 
ments by categories (e.g., a + for a strength or a - for a weakness). This 
allows participants to input comments in parallel, more rapidly than in an 
oral discussion where speakers must take turns. It also provides a written 
record of the comments. 


Figure 10.5 

MeetingWare Discuss Module 

Is this iteM= * Appropriate? Uhat are its= * Strengths? 

* Unique? # Ueaknesses? 

* Singular? 


ITEM 1 


GAS MILAGE IS CRITICAL DURING THIS PERIOD OF TIGHT BUDGETS 


Tire left: 2 Min 53 Sec 


IF13=Brealq 


ReMaining-' 2 I tens 6 Min 
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Figure 10.6 

MeetingWare Comment Module 


No, [ TOPIC; BUILD A PARKING “MEGA-STRUCTURE 11 FOR 5888 CARS 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 


1THIS OPTION WOULD BE MUCH TOO EXPENSIVE 

ift NEW PARKING STRUCTURE WILL ONLY ENCOURAGE MORE DRIUING 

iIF THIS WERE BUILT U*©ERGROUK), IT WOULD BE A GOOD SOLUTION 




List Organization 

Two modules are generally concerned with list organization: LIST EDIT 
and ORGANIZE. LIST EDIT contains a minimum set of functions for 
modifying a list of Ideas. The group can edit or delete existing ideas and can 
add new Ideas to the list. This module is appropriate when dealing with a 
short, simple list in which all ideas are on the same level of analysis. 

If the group needs to work with a more complex list of ideas where it is 
important to structure the list, the ORGANIZE module may be more appro¬ 
priate. ORGANIZE builds outlines, which is a technique for organizing 
ideas that almost everyone knows how to use (see Figure 10.7 for a typical 
screen). 

With ORGANIZE, groups can structure lists in at least three different 
ways: (1) they can group items together based on similarity; (2) they can 
place items at different levels of the outline to represent different levels of 
analysis; and (3) they can place items so as to represent temporal ordering or 
sequence. As they build the outline, it is displayed on the front screen, and 
ideas can be placed in the appropriate location in the outline following dis¬ 
cussion by the group. This can help reduce conflict arising when group 
members approach a problem from differing points of view or levels of anal¬ 
ysis (Gilbert 1978). 

The group can easily change the outline by moving, copying, inserting, or 
deleting sections. Once an outline is completed, the group can decide to ex¬ 
tract part or all of the outline, sort it, and then write it to a file. Thus, par- 
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Figure 10.7 

MeetingWare Organize Module 


Load files from disk 

HeHIHM EDIT OUTLINE ! PRINT REPORT ! SAVE FILES ! QUIT 


SB 
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+ REDUCE CAR-DRIVING 

+ Hake it easier to jet to work without a car 
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ticipants can isolate any section or level of an outline for further processing 
by other MeetingWare modules. 

When sufficient equipment is available (two PCs with video projection 
devices) it is often useful to combine the DISCUSS and ORGANIZE mod¬ 
ules in what we call the dual screen mode. In this approach, one PC runs the 
DISCUSS module, which displays the list of ideas one at a time at the front 
of the room. A second PC simultaneously runs the ORGANIZE module, 
which displays the outline at the front of the room next to the DISCUSS 
display. As each item comes up for discussion, the chauffeur enters it into 
the outline as determined by the group. 


Item Evaluation 

Groups can use several modules to evaluate items on a list: CROSS- 
IMPACT ANALYSIS, VOTE, RANK, RATE, and WEIGHTED FACTORS 
ANALYSIS. CROSS-IMPACT ANALYSIS can compare any two lists of 
items. For example, a group might want to compare solution alternatives to 
obstacles in order to indicate which solutions might help overcome which 
obstacles. Individual participants can indicate both the strength and direc¬ 
tion of an impact by using a — 5 to + 5 scale. The software creates summary 
matrices that show either the sum of the groups’ judgments or an average. 
The group can also view matrices showing the level of agreement regarding 
an interaction, by choosing to see the standard deviation or range displays. 
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All these matrices can be filtered to show only values in a desired range (e.g., 
all the high or low values). Graphs showing the specific ratings for a given 
interaction are available, which allows for an analysis of the evaluation with¬ 
out compromising the anonymity of participants. Three typical CROSS¬ 
IMPACT screens are shown in Figures 10.8, 10.9, and 10.10. 

The VOTE, RANK, and RATE modules are somewhat similar. In VOTE, 
a participant casts a vote for a single preferred item. In RANK, each partici¬ 
pant arranges the items in rank order. The RATE module allows the partici¬ 
pants to indicate their preferences on a 1-10 scale. Each module provides a 
summary of the evaluation on the shared screen, showing total or mean val¬ 
ues, and a graphic display of the distribution of votes, ranks, or ratings. A 
typical group summary screen is shown in Figure 10.11. 

The WEIGHTED FACTORS ANALYSIS module is the most complex of 
the evaluation approaches. This multiattribute tool allows the group to eval¬ 
uate up to ten alternatives at a time using up to fifteen criteria. The partici¬ 
pants can individually assign weights to each criterion, and the system will 
calculate group weights by taking the mean value. Then each participant 
rates how well each alternative meets each criterion. The product of the 
weight and rating determines the score for a given alternative on a given cri¬ 
terion. The software sums these scores across all criteria for each alternative, 
and the alternative with the highest score is considered the best. The soft¬ 
ware integrates individual evaluations into a master matrix that summarizes 
the results, using the mean of the groups’ individual scores. A typical 
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Figure 10.9 
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WEIGHTED FACTORS matrix screen is shown in Figure 10.12. Since this 
report contains such a wealth of detailed information, a set of graphics dis¬ 
plays is available as an aid to analysis. One summary display shows the final 
weighted scores as a set of bar graphs, which makes it easy to see how each 
alternative performed (Figure 10.13). 

A more detailed analysis is possible by displaying graphs showing how 
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Figure 10.11 
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Figure 10.13 

Weighted Factors Summary Graph 
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each alternative scored on each criterion. An example for the first four crite¬ 
ria is shown in Figure 10.14. This can be truly useful in helping the group 
understand why an alternative did well or poorly in the final analysis. It is 
also possible to show tables summarizing the level of agreement on the item 
ratings, as measured by the standard deviation or range of the scores (Figure 

10.15) . When the software identifies an item with a high standard deviation, 
it can display a graph that shows the specific scores assigned by the group 
while still retaining individual anonymity. This makes it possible to identify 
areas of agreement and disagreement quickly, so that the group can focus on 
important issues. 

The weighted factors analysis module includes another feature that can 
help a group understand their preference model: sensitivity analysis. It can 
often be useful to explore the impact of changes in the weights assigned to 
the criteria or the ratings assigned to the alternatives. This is especially true 
when there is no single alternative that scores well above all others or when 
there is considerable uncertainty about the appropriate weights or ratings. 
The criteria sensitivity analysis routine displays the criteria and their current 
weights and then allows the chauffeur to modify the weights and run the 
preference model again, calculating a new weighted factors matrix (Figure 

10.16) . Thus, the group can explore the effect of varying the criteria weights 
in an easy and efficient manner. Each time, the group can view the graphs 
and can print a report as a permanent record of the scenario. In a similar 
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Figure 10.16 

Sensitivity Analysis Using Criteria 
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manner, the group can modify the ratings for an alternative and see the im¬ 
pact on final weighted scores. When the sensitivity analysis is completed, the 
system can restore the original weights and ratings automatically. 


Information Sharing 

Meeting Ware provides two tools that support information sharing for the 
group. The first, VIEWER, allows the group to share information during a 
session through the shared display at the front of the room. This module will 
load any Symphony or Lotus 1-2-3 application and allow the group to view 
or manipulate the application. This could include a spreadsheet model, a 
database, a text document, or a graph. 

The second tool, REPORTWRITER, allows the facilitator to work with 
history files created during one or more MeetingWare sessions and create a 
final summary report of the group’s activities. It is essentially a word proces¬ 
sor designed to work with MeetingWare data files. 


Session Management 

Two tools help in managing MeetingWare sessions. SCRIPTWRITER 
supports the design, modification, and execution of meeting scripts (Figure 
10.17). Basically, it allows the meeting facilitator to choose the modules she 
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or he desires for a meeting and to define the purpose of each tool and the file 
names that they will use to save the data. It makes it easy to insert, delete, 
edit, or move steps in a meeting script. It then makes a meeting semiauto¬ 
matic by executing the script as defined by the facilitator. However, the script 
retains flexibility since it can be modified during a meeting, as group needs 
dictate. 

The SESSION MANAGER module helps manage GDSS environ¬ 
ments where there may be many clients and/or many facilitators using 
MeetingWare. It will help the facilitator install the correct data files for a 
session and back up the files at the end of the session. It reduces the possi¬ 
bility that files from different groups will become intermixed on the system, 
leading to confusion. 

BENEFITS OF MEETINGWARE 

MeetingWare includes a variety of features that provide important bene¬ 
fits to decision-making groups. A summary of these benefits is given in Fig¬ 
ure 10.18, organized by tool module. Let us consider these benefits. 


Anonymity 

MeetingWare provides anonymity in three critical phases of decision mak¬ 
ing: idea generation, idea discussion, and idea evaluation. During idea gen- 
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Figure 10.18 

Features and Benefits of MeetingWare 
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eration, this feature is intended to reduce conformity pressures and, thus, 
increase the likelihood that participants will contribute ideas in which they 
truly believe. It is also intended to increase the probability that participants 
will contribute creative or unusual ideas. Both the LIST and BRAIN¬ 
STORM tools provide anonymity. It should be noted that nothing prevents a 
participant from revealing their authorship of an idea if they so desire, and 
such behavior has not been unusual. The GDSS chauffeur is trained to dis¬ 
courage participants from trying to guess authorship of others’ ideas or 
from engaging in any behaviors designed to pressure others to reveal then- 
authorship. 

The COMMENT tool allows for anonymous written discussions. While 
this can be a very useful feature, we do not believe it should always replace 
the alternative approach, DISCUSS, which allows for face-to-face oral dis¬ 
cussions. However, when there are valid reasons for anonymity, COM¬ 
MENT will provide it. 

During idea evaluation, anonymity is intended to protect participants’ pri¬ 
vacy so they feel free to express their true judgments concerning the worth of 
contending ideas. All the evaluation modules allow for an anonymous ex¬ 
pression of individual judgments. While participants are also free to reveal 
their evaluations, this has rarely occurred. As in the idea generation phase, 
participants are discouraged from pressuring others to reveal their evalua¬ 
tions. 


Focused Discussions 

Groups frequently experience problems staying focused on the task at 
hand; It is all too easy to wander off on tangential issues. In addition, when 
multiple ideas need discussing, groups frequently spend disproportionate 
amounts of time on the first few ideas discussed, while virtually ignoring 
Ideas at the bottom of the list. Four tool modules help groups overcome 
these problems: DISCUSS, COMMENT, LIST EDIT, and ORGANIZE. 

By presenting lists of Ideas one at a time on a common screen, the DIS¬ 
CUSS and COMMENT modules focus attention on the current idea. DIS¬ 
CUSS helps to even out the time spent discussing each idea by using a timer. 
The focus around the common screen appears to make it easier for members 
to give up their personal ownership of an idea and seems to result in a sense 
of group ownership of these ideas. The LIST EDIT and ORGANIZE mod¬ 
ules also tend to focus groups around the shared tasks of revising and struc¬ 
turing a list. 


Parallel Input 

One of the limitations of verbal discussions for idea generation is the re¬ 
quirement that only one person talk at a time. Several MeetingWare tool 
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modules allow all participants to enter information simultaneously: this in¬ 
cludes both of the idea generation tools, the COMMENT tool, and all the 
evaluation tools. This timesaving feature assumes that the system is running 
in a full network configuration. Recent GDSS field studies (Keleman, Gar¬ 
cia & Lewis 1989; Lewis & Keleman 1990) report that participants experi¬ 
enced gains in meeting efficiency. 


Report Generation 

Groups frequently rely on a single member to keep track of actions and 
decisions in the group and to record them in the minutes. This approach can 
lead to problems of accuracy, completeness, timeliness, and potential bias. 
All the MeetingWare modules except DISCUSS can automatically produce 
reports summarizing group activities. Typical reports include lists, outlines, 
comparison matrixes, and evaluation results (numeric and graphic summa¬ 
ries). Unlike many meetings where members may wait days or weeks for 
minutes and meeting reports, MeetingWare reports are available immedi¬ 
ately, without any significant delay. 


Text Manipulation 

When groups use chalkboards and flipcharts, it is often difficult and 
messy to edit and manipulate text. Four MeetingWare tools allow partici¬ 
pants easily to edit their ideas to correct mistakes or clarify meanings. While 
ideas are first being entered (in LIST, BRAINSTORM, or COMMENT), 
participants have access to a set of line editing commands to insert, delete, 
or modify text. In addition, when the whole group is considering lists of 
ideas (in LIST EDIT and ORGANIZE), they have access to a similar set of 
editing commands to make agreed-upon changes. The result is often a well- 
constructed list of clearly stated ideas, without messy corrections. 


List Structuring 

There is another form of text manipulation that is even more difficult with 
chalkboards and flipcharts: organization or structuring of lists of ideas. If 
idea generation always resulted in ideas expressed at the same level of analy¬ 
sis, with a single topic focus, then list organization might not be necessary. 
This is seldom the case. Commonly, groups generate lists of ideas expressed 
at different levels of analysis and/or centered around different though re¬ 
lated topics. It is often confusing to try to deal with these items on a single 
list, and it can be impossible to evaluate such a diverse list meaningfully. 

MeetingWare provides a means to handle this problem through the 
ORGANIZE module. An outline can represent the structure of the ideas 
generated by the group. The manipulation of this outline is simple and 
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straightforward, compared to the manipulation of flipcharts or chalkboard 
diagrams. After a group successfully organizes a list, they can extract sec¬ 
tions or levels of the outline for separate evaluation, thus avoiding a con¬ 
founding of levels or topics. 

When a group uses COMMENT along with ORGANIZE, they can merge 
the typed lists of comments with the original list of topics automatically. If 
they entered codes during the COMMENT session, the ideas will be grouped 
by their codes In the outline under the appropriate topic. 


Preference Analysis 

For some evaluation methods, it may be useful for the group to review the 
pattern of individual judgments without being able to connect specific indi¬ 
viduals with their evaluations (i.e., anonymity is preserved). This may help 
identify errors in evaluation (such as scale reversal) and/or areas of consen¬ 
sus and disagreement. The CROSS-IMPACT, RANK, RATE, WEIGHT, 
and WEIGHTED FACTORS modules provide a means for such analysis of 
preference patterns. The group will be able to review the ranks or ratings 
given to each item but will not know who assigned them. MeetingWare 
chauffeurs are instructed to look for certain patterns that may indicate a 
need for discussion. These modules allow the participants to modify their 
most recent set of evaluations following such a discussion. 

CROSS-IMPACT and WEIGHTED FACTORS ANALYSIS include addi¬ 
tional features for preference analysis. Both contain tables summarizing the 
standard deviation and range of the group’s scores, as well as filters for 
quickly identifying areas of agreement and disagreement. 


Reduced Computational Overhead 

Computers are well known for their ability to perform rapid mathematical 
calculations. MeetingWare takes advantage of this feature for the combina¬ 
tion of individual preferences into group preferences. Although the mathe¬ 
matics involved are not especially complex, a large number of operations 
can be involved in such preference calculations, and manual methods can be 
slow, tedious, and error prone. Indeed, although methods such as weighted 
factors analysis are superior for groups coping with complex decisions, these 
techniques are often avoided because they are so difficult to use in a manual 
setting. The evaluation modules in MeetingWare reduce the manual compu¬ 
tational overhead to zero and may result in a greater willingness to use more 
sophisticated decision making methods. 


Rapid Sorting 

It is frequently convenient to work with lists that have been meaningfully 
sorted. A number of MeetingWare modules can sort lists, either alphabeti- 
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cally or by an evaluation value such as rank or rating. Although it is not 
impossible to sort material manually, the ease with which a computer sorts 
material makes it a more attractive option from a practical point of view 


EXPERIENCES WITH MEETINGWARE APPLICATION 

Several laboratory and field studies have been conducted with Meetins- 
Ware and its predecessor, FACILITATOR. The first laboratory study was 
carried out in 1982 and is described in detail elsewhere (Lewis 1987). This 
study found evidence that a GDSS can improve both the product of group 
problem solving (solution alternatives) and the group process itself (partici¬ 
pation, creativity). It also found that the group members accepted the use of 
a GDSS. 

After this initial study, the FACILITATOR software was completely re¬ 
written, and another laboratory study was carried out in 1985. Evidence was 
found that this GDSS reduced conflict, brought order to the discussions, 
encouraged equality of participation, and did not reduce interpersonal sup¬ 
port in the groups. The studies provided a great deal of useful information 
about GDSS design issues, many of which are discussed in Lewis and Kele- 
man (1988). In 1986-1987 FACILITATOR was used to support a county¬ 
wide group of twenty-two local public and not-for-profit organizations 
charged with coordinating economic planning efforts. As one pan of a 
larger planning process, representatives from these organizations were 
placed into three groups that used a GDSS approach to consider objectives, 
obstacles and constraints, and action strategies. The process resulted in the 
first agreed-upon coherent framework for guiding economic development in 
the county. A postsession questionnaire indicated that participants felt the 
GDSS improved the efficiency, effectiveness, and overall productivity of the 
groups. They felt that there was far more equality of participation than in 
their experience with ordinary groups. All of the participants said they 
would like to use a GDSS again. (This study is described in greater detail in 
Lewis, Keleman & Garcia 1988.) 

Following this field study, the FACILITATOR system was decomposed 
into a set of smaller modules, renamed MeetingWare. This system has been 
used in another field study over the last three years. A human resources divi¬ 
sion of a regional university decided to use a GDSS in support of its yearly 
planning efforts. Over the three-year period, there has been a noteworthy 
impact on the quality of the yearly plan. Goals are better articulated and 
more specific, with measurable outcomes. Constraints and resources are 
now clearly identified and associated with relevant goals. A greater number 
of action strategies have been identified for each goal. Through a question¬ 
naire, participants have indicated they feel the GDSS improved the produc¬ 
tivity, efficiency, and effectiveness of the group, and all participants said 
they would use a GDSS again and would recommend it to others. (A com- 
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plete description of this study can be found in Keleman, Garcia Sc Lewis 
1989.) 

A recent study using MeetingWare was carried out at Curtin University of 
Technology in Perth, Western Australia. Two aspects of GDSS use were stud¬ 
ied: perceptions of professional facilitators concerning GDSS impacts and 
impacts of GDSS on the workloads of chauffeurs, facilitators, and group 
participants. An initial report regarding the first aspect can be found in 
Lewis and Whiteley (1992); the second is described in Pollock, Atkinson, 
and Lewis (1992). 

FUTURE PLANS 

Like many other current software products, MeetingWare is being modi¬ 
fied and improved in a continuous development cycle. In its current form, it 
includes somewhat more than the minimum set of modules necessary to be a 
useful GDSS. Over the next several years, we anticipate adding a number of 
new features to enhance its usefulness. For example, we are working on a 
version for Microsoft Windows. We are also exploring the possibility of 
opening the MeetingWare platform to others who may want to develop spe¬ 
cialized modules to add to the library. Some of the specific modules we an¬ 
ticipate adding are discussed below. 


Additional Evaluation Tools 

A number of techniques have been described for evaluation of group pref¬ 
erences that are more sophisticated than those currently provided by 
MeetingWare (Raiffa 1970; Sage 1977; Saaty 1986). It is likely that one or 
more of these will be added as tool modules in the next few years. A signifi¬ 
cant issue is whether these tools can be presented in such a manner that 
groups without technical backgrounds will be willing and able to use them. 


Graphics 

Some concepts are best expressed visually, and some group participants 
are likely to ‘ihink visually” more than others. We would like to offer a 
graphics-oriented tool module in MeetingWare. Such a tool could be used to 
draw sketches, flowcharts, diagrams, and so on. This can be especially use¬ 
ful for exploring the organization and structure of problems. A formal tech¬ 
nique for problem representation such as consensus mapping (Hart et al. 
1985) might be a useful addition to a GDSS tool kit. 

Asynchronous Support 

While our primary 7 interest has been in face-to-face groups, we realize that 
on many occasions it is difficult and/or unnecessary to gather an entire 
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oxip together at one time in one place. Several of the Meeting Ware tools 
mid. be modified to allow for asynchronous meetings. Primary candidates 
ight be the LIST tool and all the evaluation tools. Thus, participants could 
nerate and enter their ideas at their own convenience and then meet as a 
hole group for discussion and list organization. Finally, an evaluation 
lase could take place over a one- or two-day period as each member had 
ne to use the system to enter judgments. The entire group could meet 
;ain to review evaluation results and revote if desired. 

Process Monitor 

It may be useful for a group to have real-time assessments of participant 
elings about the group process. This could include issues such as need for 
eaks, relevance of current activities, levels of conflict, agenda setting, and 
1 on. A GDSS might include a simple pop-up polling mechanism to allow 
r these process checks. In some cases, a continuous graphic display might 
* the best approach (e.g., a “conflict meter”). 

Knowledge-Based GDSS Adviser 

As more and more Meeting Ware tools become available, it becomes more 
fficult to choose the appropriate tools and combine them in me anin gful 
quences, especially for novices. We have developed a prototype of a 
lowledge-based system that gives advice to potential Meeting Ware users. It 
dps to determine whether a GDSS approach appears appropriate for their 
:uation, whether they have the necessary resources to proceed, and what 
ol modules they should consider. Ultimately, we anticipate expanding the 
pabilities of this system so that it can write complete Meeting Ware scripts 
r GDSS users. 

JMMARY 

Meeting Ware is an example of a new class of software designed to support 
icision making in small face-to-face groups. It provides a set of modular 
ols that can be flexibly combined in various patterns to meet the unique 
ieds of specific groups. Initial laboratory and field studies indicate that 
vision support systems such as Meeting Ware can improve both the process 
id product of small group decision making. New capabilities planned for 
e near future should increase the effectiveness of this GDSS software. 
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Utilization of Decision-Aiding 
Software in the U.S. Government 


Fred Wood 


Advances in information technology are fueling a revolution in computer 
modeling, both inside and outside government. The 1980s and early 1990s 
have been characterized by the expansion of computer modeling, via low- 
cost microcomputers and user-friendly software, literally into the office of 
the individual scientist, engineer, analyst, or manager, and, simultaneously, 
via supercomputers, to the new limits of modeling complexity demanded in 
the scientific, energy, space, climate, and defense sectors. The span and di¬ 
versity of computer modeling activities in the federal government have never 
been greater. About 60 percent of federal agency units responding to an Of¬ 
fice of Technology Assessment (OTA) survey reported at least some use of 
computer modeling, with the number of applications ranging up to two 
thousand per agency component. 

The use of computer-based decision analytic techniques has also increased 
dramatically. Such techniques typically include computer software that can 
help decision makers or staff analyze a specific problem, possible decision 
options, and the likely or possible consequences. About 90 percent of fed¬ 
eral agency units report use of spreadsheet software, about one-half use 
quantitative decision techniques (e.g., linear programming), about one- 
fourth use forecasting and qualitative techniques (e.g., decision trees), and a 
handful use decision conferences and computer conferencing. 

Overall, executive branch officials responding to the OTA survey believe 
these techniques to be very useful, even essential, to agency decision mak¬ 
ing. However, few can document this claim, other than by citing examples, 
because there has been little research on the impact of decision support tech¬ 
niques on agency decision making. The limited research that is available, 
primarily by academics on model implementation, suggests that models 
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(and, by extension, other decision analytic techniques) can and do have a 
significant impact on agency decision making. Modeling may become a sig¬ 
nificant element in the process of negotiation over assumptions and options 
that is an integral part of agency (and, in general, political) decision making. 
However, models can be wrong, and models can be misused. 

Several possible actions could help improve sharing of expertise and learn¬ 
ing; facilitate public and congressional access where appropriate; enhance 
congressional and public understanding of the strengths and limitations, 
uses and abuses of modeling; and improve the government’s return on a sig¬ 
nificant investment: 

• Establishing guidelines or standards for model documentation, verification, and 
validation. 

• Establishing directories of major modeling applications. 

• Clarifying procedures on public access to modeling details. 

• Conducting further research on the impact of computer modeling and decision 
support on agency decision making. 

• Conducting basic and applied research on modeling and decision support metho¬ 
dologies. 

• Conducting further testing and development of the decision conference technique. 

• Bringing computer modeling and decision support clearly within the scope of in¬ 
formation resources management. 

After a review 7 of the entire range of known applications of information 
technology in the federal government, computer-based modeling and deci¬ 
sion support stood out as application areas about which little concrete 
information was available. There was no current, reliable source of informa¬ 
tion on federal government use of computer-based modeling and decision 
support. In order to develop a sound basis for understanding trends and 
issues relevant to computer modeling, this topic was included in an OTA sur¬ 
vey sent to all thirteen cabinet departments and twenty independent agen¬ 
cies. 

For purposes of this survey, computer modeling included the entire range 
of mathematical models used to support agency activities and programs — 
from small models run on microcomputers in individual offices to large, 
complex models ran on supercomputers. A model is an abstraction, analog, 
image, or representation of some aspect of current (or future) reality rele¬ 
vant, in this case, to the missions and programs of federal agencies. Ail but 
the very simplest mathematical models are now routinely programmed as 
sets of equations and run on computers. Thus, most models are computer- 
based models (computer models, for short). Computer models can be used 
for a variety of purposes, from conducting scientific research in aeronautics 
or climate, to engineering the design of a new highway bridge, to estimating 
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future numbers of school-age children, to analyzing the fiscal impacts of 
alternative Medicare reimbursement policies. Computer models can be and 
are used to support agency decisions but have many other purposes as well. 

Consideration of computer-based decision support included several types 
of analytical techniques (along with the necessary computer software, hard¬ 
ware, data sets, graphic displays, and the like) used to support or assist deci¬ 
sion makers. The categories of computer-assisted analytical techniques used 
in the OTA survey and in this chapter are: 

• Spreadsheet computer software. 

• Forecasting techniques (e.g., regression analysis, Delphi survey). 

• Quantitative decision analytic techniques (e.g., linear programming, queuing anal¬ 
ysis, systems analysis, critical path analysis). 

• Quantitative decision analytic techniques with judgmental input (e.g., decision 
trees, subjective probability, multiattribute utility). 

• Decision conference techniques (e.g., interactive use of computer-assisted analyti¬ 
cal techniques by decision makers in a group situation). 

• Electronic voting techniques (e.g., consensor, computer polling). 

• Computer conferencing for decision analysis. 

• Other (e.g., expert systems). 

Most of these techniques also involve the use of models. For example, an 
analysis of the relationship between rainfall, temperature, and crop yield 
might use a computer-based multiple regression model to understand the 
performance of different varieties of crops under various climatic conditions 
or to help an agricultural extension agent or the Agency for International 
Development agricultural employee select specific varieties to recommend 
for spring planting. 

This chapter presents OTA’s findings on key trends and issues relevant to 
computer modeling and decision support. 

KEY TRENDS 

Information Technology Fueling Modeling Revolution 

Several key technological developments have profoundly changed the con¬ 
duct of analytical, forecasting, and research activities that utilize modeling. 
The first is the microcomputer revolution. From almost no microcomputers 
twelve years ago, federal agencies now have, collectively, more than 500,000. 
Access to computer power truly has been decentralized, both in terms of 
actual desktop computer capability and the use of microcomputers as access 
points to larger mainframe computer resources. This phenomenon parallels 
that found in the research and business communities outside government. 
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A second key trend is the large increase in user-friendly computer soft¬ 
ware, especially software suitable for microcomputers. This includes a wide 
range of spreadsheet, modeling, and decision analytic software that permits 
many small-scale, relatively simple decision analytic and modeling applica¬ 
tions. 

A third key technological trend is at the high end of computer power —the 
supercomputer. Supercomputers are extending the limits of modeling com¬ 
plexity, from aerodynamics to climate. In the United States, supercomputers 
have been installed at, for example, the Lawrence Livermore Laboratory 
(Department of Energy, DOE) for magnetic fusion research and the Ames 
Research Center (National Aeronautics and Space Administration, NASA) 
for numerical aerodynamics modeling. Both NASA and DOE officials have 
found that supercomputers are essential to their modeling activities. 

Use of supercomputers is not limited to government agencies. For exam¬ 
ple, with National Science Foundation (NSF) funding, additional supercom¬ 
puter centers have been established at several universities. At the University 
of Illinois, illustrative applications range from high-energy physics (e.g., 
simulation of a particle accelerator to test theories about elementary parti¬ 
cles), to chemistry (e.g., simulation of molecular behavior), to civil engi¬ 
neering (e.g., modeling of transportation systems in the Chicago area), to 
physiology* and biophysics (e.g., modeling of electrical activity of nerve and 
muscle ceils). 

The earliest computer modeling dates back to the 1950s when first-gener¬ 
ation computers were used, for example, to run simple numerical models for 
weather prediction. Until around 1970, federal government modeling was 
concentrated in the scientific, energy, space, and defense sectors — sectors 
with the greatest computational needs and the resources to pay for the 
expensive but necessary computer power. During the 1970s, however, the 
widespread availability of relatively cheap computers contributed to the ex¬ 
pansion of computer modeling activities to areas such as air pollution, water 
resources, solid waste management, urban development, and transporta¬ 
tion. The 1980s and early 1990s have been characterized by the expansion of 
computer modeling, via low-cost microcomputers, literally into the office of 
the individual scientist, engineer, analyst, or manager and simultaneously 
via supercomputers, to the new limits of modeling complexity demanded in, 
for example, the energy and climate sectors. 1 The results of OTA’s survey 
Indicate that the span and diversity of computer modeling activities in the 
federal government have never been greater. 

Weather and climate modeling is a good illustration of how computer 
modeling in general has essentially developed in parallel with advances in 
computer power. The record shows that the complexity of weather and cli¬ 
mate models quickly expands to push the limits of the computational power 
and capacity of each successive generation of computer technology. 2 
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Continuing Heavy Federal Use of Computer Modeling 

Federal agency use of computer modeling is substantial. Almost 60 per¬ 
cent of 141 agency components responding to the OTA survey reported some 
use of computer modeling to support agency activities and programs, and 
this excludes use of decision analytic techniques such as spreadsheet soft¬ 
ware. (The OTA survey was limited to the federal executive branch. Other 
OTA research reviewed use of computer modeling by congress 3 and state leg¬ 
islatures. 4 ) 

For agencies that could estimate the total number of modeling applica¬ 
tions, the number ranged up to two thousand per agency component. 
Among the heaviest reported computer model users are the Economic Re¬ 
search Services (Department of Agriculture), Office of Program Analysis 
and Evaluation (Department of Defense, DOD), U.S. Geological Survey 
(Department of the Interior), Federal Highway Administration (Department 
of Transportation, DOT), and the Nuclear Regulatory Commission (NRC). 

OTA asked agency components to list the ten heaviest areas of modeling 
application. The results demonstrated the wide diversity in the purposes for 
which computer modeling is used by federal agencies. Examples from seven 
selected agencies are shown in Table 11.1. 

Although the results of the OTA survey are not adequate to make a precise 
estimate of the number of modeling applications, it is clear that the total is 
far higher than previously thought. A 1982 General Accounting Office 
(GAO) survey identified 357 models used in the agency policymaking 
process, based on responses from twelve of the thirteen cabinet departments 
and eighteen independent agencies. 5 The GAO survey very likely under- 
reported the total number of policy-relevant models as of that time (1982), 
and the number has probably increased since then. While a precise estimate 
is neither possible nor necessary, the ballpark minimum would appear to be 
in the thousands for policy models and tens of thousands for all types of 
computer models used by federal agencies. The numbers could be much 
higher, especially if spreadsheet-type models are included. 


Rapidly Increasing Federal Use of Computer-Based Decision 
Support and Analysis 

Computer-based decision analysis dates back to the 1960s for its theoreti¬ 
cal roots (e.g., as developed by Howard Raiffa of Harvard University) 6 and 
to the 1970s for its practical development and early application, primarily in 
the military and business sectors. Early federal government sponsors of re¬ 
search and development (R&D) on decision analysis included the Defense 
Advanced Research Projects Agency and the Office of Naval Research. The 



Table 11.1 

Federal Agency Modeling Applications 


Economic Research Service (Department of Agriculture) 

An estimated 2,250 computer modeling applications, 
including: 

• analysis of farm program alternatives 

• analysis of world food supply, capacity, and response 

• analysis of conservation alternatives 

• trade policy analysis 

• forecasting of commodity supply and demand 

Forest Service (Department of Agriculture) 

An estimated 100 applications, including: 

• timber resource allocation model 

• integrated pest impact assessment system 

• forest growth and yield analysis 

• fire management and planning model 

• engineering design models for roads, structures, and 
buildings 

Office of Secretary of Defense (Office of Program Analysis 
and Evaluation} 

An estimated 1,250 applications, including: 

• impact of defense spending on U.S. economy 

• strategic defense initiative effectiveness studies 

• military force mobility modeling 

• impact of procurement schedule changes on acquisition 
costs 

• impact of second-source/competitive procurement on 
acquisition costs 

Joint Chiefs of Staff (Department of Defense) 

A large number of applications, including: 

• strategic nuclear war plans analysis 

• non-strategic nuclear force mix analysis 

• military force posture analysis 

• improving crisis war planning processes 

• nuclear damage assessment 

Bureau of Indian Affairs (Department of the Interior) 

An estimated 15 applications, including: 

• road and bridge design 

• forest and range fire risk analysis 

• rangeland usage and conditions analysis 

• rangeland market appraisal 

• oil and gas lease management and planning 
Office of Assistant Secretary for Program Evaluation 

(Department of Health and Human Services) 

A small number of applications, including: 

• revenue impact analyses of, for example, including so¬ 
cial security and welfare benefits in taxable income, 
providing additional tax exemptions for children in the 
first year after birth, and replacing Federal income tax 
credits for the elderly with higher deductions. 

• estimates of participation rates for Aid for Dependent 
Children (AFDC) recipients in the Food Stamp Program. 

• estimates of the Deficit Reduction Act impact on AFDC, 
Food Stamp, and Supplemental Security Income bene¬ 
ficiaries. 

Federal Emergency Management Agency 
An estimated 100 applications, including: 

• mobilization for nuclear and general war 

• earthquake damage and economic impact estimates 

• residual capacity of U.S. economy after nuclear war 

• strategic stockpile policy development 

• flood damage analysis 

Source: Office of Technology Assessment Federal Agency Data Request. 



Utilization of Decision-Aiding Software 


239 


early decision analytic tools were implemented with paper and pencil, slide 
rule, and/or calculator. 

Since decision analysis techniques may involve many options (e.g., nu¬ 
merical probabilities based on empirical evidence and/or quantified judg¬ 
ments of uncertain future events), the number of calculations per run can be 
large, and the typical application involves many runs with changing options 
and values. Thus, decision analysis is a natural match with electronic com¬ 
puter capability. Therefore, almost all decision analytic techniques are sig¬ 
nificantly, if not entirely, run on computers, at least for the computational 
aspects. Many decision analysis software packages are now available off the 
shelf for use on microcomputers, and the software and the hardware, to¬ 
gether with relevant databases, are frequently known as decision support 
systems. 

The results of the OTA survey provided a good profile of agency use of 
decision analytic techniques —the first complete profile known to exist. The 
results are likely to understate the full extent of use, given the highly decen¬ 
tralized nature of decision support. Nonetheless, the results are generally 
consistent with the perceptions of informed observers, especially with re¬ 
spect to the relative differences in levels of use for the various techniques. 

The results are summarized in Table 11.2. Spreadsheet software is used by 
almost ail (88 percent) of the agency components responding, and half of 
the remaining agency components (eight of sixteen) are planning to use 
spreadsheet software. Almost half (47 percent) of agency components re¬ 
port the use of quantitative decision analytic techniques, with another thir¬ 
teen agency components planning to use such techniques. About one-fifth 
(22 percent) of agency components report the use of quantitative decision 
analytic techniques with judgmental input, and about one-fifteenth report 
the use of decision conference techniques. Nine agency components report 
the use of decision conferences, and another seven components indicate that 
they are planning to do so. About one-twentieth report the use of computer 
conferencing for decision support, and two agency components indicate the 
use of electronic voting techniques. Also, three components report the 
planned use of expert systems or artificial intelligence for decision support. 

The use of spreadsheet software is spread throughout all agencies, and the 
use of quantitative techniques is fairly widespread in, for example, the De¬ 
partments of Agriculture, Commerce, Defense, Interior, Transportation, 
and Treasury, and about two-thirds (twelve of nineteen) of the independent 
agencies surveyed. However, DOD is the only agency with more than half of 
agency components reporting use of quantitative decision analytic tech¬ 
niques with qualitative input (e.g., decision trees, multiattribute utility). 
DOD is the only agency reporting significant use of decision conferences 
(about one-third of DOD components reporting), although there was scat¬ 
tered, infrequent use reported in Agriculture, Interior, and Transportation. 

With respect to use of quantitative decision analytic techniques, the Inter- 
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Federal Agency Current and Planned Use of Computer-Assisted Decision Analytic Techniques 
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national Economic Policy (IEP) Group of the International Trade Adminis¬ 
tration (Department of Commerce) is illustrative. This agency component 
combines the use of decision analytic techniques, models, and databases to 
“help improve decisionmaking” and “to enhance lEP’s ability to provide 
policymakers and U.S. business with comprehensive information on trade 
and investment matters generally.” As one other agency example, the Drug 
Enforcement Administration (DEA) (Department of Justice) is planning to 
use quantitative decision techniques to optimize allocation of agency re¬ 
sources (agents, monies for purchase of information and evidence, etc.) in 
terms of productivity as measured, for example, by the number of repeat 
offender arrests, volume and value of drug interdictions, and reductions in 
drug availability. Also, DEA plans to use quantitative techniques with judg¬ 
mental input and artificial intelligence techniques for investigative and intel¬ 
ligence purposes. 

Other examples of the use of decision analytic techniques, especially those 
combining quantitative and qualitative (judgmental) methodologies, in¬ 
clude: 

• DOD use of multiattribute utility analysis to aid in the evaluation and acquisition 
of major military systems such as the advanced scout helicopter, light armored 
vehicle, mobile protective weapons system, and single channel ground and air¬ 
borne radio system. 

• Defense Nuclear Agency use of multiattribute utility and cost-effectiveness analy¬ 
sis to aid in R&D budgeting. 

• Department of the Air Force use of decision analytic techniques to aid in planning 
and targeting air strikes against enemy air bases and in developing command, con¬ 
trol, and communication countermeasures. 

• NRC use of decision analysis to aid in the evaluation of proposed new regulatory 
requirements and safeguard designs. 

• DOE use of decision analysis to aid in implementation of the Nuclear Waste Policy 
Act of 1982 and the siting of repositories for high-level nuclear waste. 

• National Security Council use of decision analysis in evaluating alternative strate¬ 
gies for the Middle East. 

• President’s Council on International Economic Policy use of decision analysis in 
evaluating alternative export control policies for computer technology. 


KEY OPPORTUNITIES FOR ACTION 
Guidelines or Standards for Model Evaluation 

Efforts to manage computer modeling and to establish some minimum 
level of standards have always lagged behind the actual level of applications 
by many years. In the 1970s, as computer modeling applications proliferated 
throughout the federal government, the National Bureau of Standards 
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(NBS) (now the National Institute of Standards and Technology), Energy 
Information Administration (EIA), and the GAO took the lead in attempt¬ 
ing to bring some coordination and coherence to civilian modeling activities. 
The Joint Chiefs of Staff (JCS) did similarly for defense modeling. 

GAO issued reports in 1976, 1978, and 1979, and NBS issued reports in 
1979 and 1981 (with EIA support). 7 A central theme in all of these reports 
was the need to develop a common framework for model evaluation or as¬ 
sessment. Many suggestions were made, but none was adopted on a govern¬ 
ment-wide basis. A very few individual agencies, such as EIA, eventually 
adopted some variant of a model evaluation procedure. 

Given the extensive use of computer modeling by federal agencies, the 
level of formal model documentation, verification, and validation appears 
to be deficient. Clearly, computer models are judged to be important by 
many federal agencies and are used for purposes ranging from research to 
decision support. However, the research on computer modeling makes two 
things abundantly clear: models can be wrong, and models can be misused. 8 
For these reasons alone, minimum modeling guidelines or standards appear 
to be needed. In addition, such guidelines presumably would make it eas¬ 
ier to strengthen the federal modeling expertise and perhaps achieve a higher 
return on what must be a substantial federal investment. (OTA did not de¬ 
velop data on the costs of modeling, and most agencies are unable to esti¬ 
mate such costs readily.) 

Some agencies (e.g., NBS, EIA, JCS) have made a concerted effort to 
develop and/or apply modeling guidelines. A lead role could be assigned to 
one of these agencies, perhaps NBS, or to one civilian and one military 
agency (e.g., NBS and JCS), for developing and promulgating a set of 
modeling guidelines. Much of the groundwork has already been done, and 
development of guidelines should be straightforward. 9 The lead agency pre¬ 
sumably would involve all major modeling agencies in the guidelines devel¬ 
opment process. Guidelines for the major, expensive, complex computer 
models would logically be more complete and extensive than guidelines for 
small, simple, inexpensive desktop models. Computer modeling could be 
brought clearly within the purview of the information resources manage¬ 
ment concept, through appropriate amendments to the Paperwork Reduc¬ 
tion Act if necessary. 


Directory of Modeling Applications 

Prior studies of computer modeling in the federal government have gener¬ 
ally concluded that directories of modeling applications would be helpful, at 
least for the major models. This possibility was reiterated in a 1982 OTA 
study on water resources models. 10 Given the extremely large number of ap¬ 
plications, a comprehensive directory would appear to be costly and diffi¬ 
cult to prepare, and many of the applications may not warrant the effort. 
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However, there is a stronger argument for a comprehensive directory of se¬ 
lected major models and for an index or pointer system to a larger number 
of other significant models and modelers, perhaps indexed by subject matte- 
and type of model. These actions would be intended to help reduce possible 
excessive overlap and duplication, encourage exchange of modeling infor¬ 
mation among modelers, and facilitate a greater degree of public knowledge 
of and access to federal modeling. Some argue that modelers in anv given 
area already know or can learn what they need to know about relevant mod¬ 
eling activities without the help of modeling directories. But given the num¬ 
ber and diversity of modeling applications, this could be difficult 
Of eighty-two agency components that reported use of computer models, 
sixteen or about one-fifth indicated the existence of a modeling directory 
Those agencies are: 

Department of Agriculture: Economic Research Service and Forest Service. 
Department of Defense: Joint Chiefs of Staff and Defense Contract Audit Aceney. 
Department of Energy: Energy Information Administration. 

Department of the Interior: Minerals Management Service, US. Geological Survey, 
and Office of Surface Mining Reclamation and Enforcement. 

Department of Justice: Justice Management Division. 

Department of Labor: Bureau of Labor Statistics. 

Department of Transportation: National Highway Traffic Safety Administration 
(NHTSA), Federal Highway Administration, and Federal Aviation Administra¬ 
tion. 

Nuclear Regulatory Commission. 

Arms Control and Disarmament Agency 
Federal Emergency Management Agency. 

Most of these directories are reported to be in paper format, although the 
Forest Service and NHTSA indicate that their directories are in an on-line 
electronic format. Also, the EIA model director} 7 is in both computerized 
and printed formats. 

In addition, some of these agency components report that they' also have a 
central reference point—-usually a designated person —with current informa¬ 
tion about modeling applications. Several other agency components that do 
not have a directory do claim to have a contact person. Among the latter 
agencies are the Defense Advanced Research Projects Agency, Defense 
Communications Agency, DOE agency wide (National Energy Software 
Center—a full clearinghouse operation, not just a person). Employment 
Standards Administration (Labor), Urban Mass Transit Administration 
(DOT), and Comptroller of the Currency (Treasury). 

In total, a little more than one-third (thirty-one of eighty-two) ot the 
agency components that use computer modeling report having a model di- 
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rectory and/or a designated contact person or, rarely, an actual clearing¬ 
house. This one-third includes many of the agency components that appear 
to be among those with the heaviest modeling activity. These figures do not 
include model directories or clearinghouses that include federal agency 
models but are maintained by nonfederal entities (e.g., universities, profes¬ 
sional associations, and private information companies). 

With respect to decision analytic support, only five agencies reported a 
directory or clearinghouse of such applications. The decentralized and 
small-scale nature of most decision analytic applications probably makes a 
directory to these techniques unrealistic. However, for major modeling ap¬ 
plications—such as the major energy, agriculture, water, transportation, and 
climate models —a directory appears to make sense. Such a directory or fam¬ 
ily of directories should be useful to all parties concerned —Congress, the 
public, agency modelers, researchers, and the like. Several prototypes exist. 
The directories would logically be computerized, to facilitate easy updating, 
and could be available in on-line electronic format as well as in paper and 
microform. A common table of contents would be helpful and presumably 
would be consistent with whatever modeling guidelines may be developed. 
The directories could be organized by, at a minimum, agency, subject area, 
and type of modeling application (e.g., scientific research decision support, 
program implementation) to facilitate easy reference. 

A lead agency could be designated, perhaps the NBS, to study the options 
and develop a feasible directory design. The modeling directories and con¬ 
tact persons reported to OTA by the agencies should provide a good base 
from which to start. 


Clarified Procedures on Public Access to Modeling Details 

Only about one-tenth of agency components using computer modeling 
have formal procedures or policies (beyond the Freedom of Information 
Act) on the availability of modeling details (e.g., structure, assumptions, in¬ 
put data) to the public and Congress, and there is wide variability among the 
procedures and policies that do exist. The overall results indicate that most 
agencies have not given much attention to questions of public and congres¬ 
sional access to model details. Some agencies cite the Freedom of Informa¬ 
tion Act as the guiding policy; others state that modeling details would 
probably be provided if sought by Congress. The following agencies indi¬ 
cated the existence of procedures or policies on the availability of model de¬ 
tails to the public and/or Congress: 

• Economic Research Service (ERS) (Agriculture). 

* Bureau of the Census (Commerce). 

• Joint Chiefs of Staff (DOD). 

* Energy Information Administration (DOE). 
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• U.S. Geological Survey (Interior). 

• Bureau of Labor Statistics (BLS) (Labor). 

• Urban Mass Transportation Administration (Transportation). 

• Federal Aviation Administration (Details). 

• Nuclear Regulatory Commission. 


Most of the major federal statistical agencies are included in this list because 
they use models in developing statistical trends and forecasts and because 
there is a highly visible public demand for their information products. Thus, 
there is a strong felt need to develop explicit access policies. 

Even among the few agencies that have explicit policies, however, there is 
considerable variability in the level of public documentation that is routinely 
made available. This does not appear to reflect necessarily an agency judg¬ 
ment to withhold certain kinds of modeling information but appears to be 
more a reflection of the particular approach selected for model documenta¬ 
tion. Examples from three agencies are presented in Table 11.3. 

The El A public documentation of major models is one of the most exten¬ 
sive of all agencies responding to the OTA survey. This is partly attributable 
to the high visibility of energy modeling, periodic concerns raised about the 
quality of EIA energy models and projections, and congressional and statu¬ 
tory requirements. For example, EIA has a statutory mandate to ensure 
“that adequate documentation for all statistical and forecast reports pre¬ 
pared ... is made available to the public at the time of publication of such 
reports.” 11 Since many such EIA reports are based on computer models, the 
models themselves are required to be documented. EIA has issued two or¬ 
ders that specify the format and public availability of model documenta¬ 
tion. 12 In addition, in part in response to congressional criticism and outside 
audits and evaluations, it appears EIA has made significant progress in doc¬ 
umenting the thirty-three major computer models currently in use. 13 EIA 
has made extensive use of model evaluations conducted by outside groups, 
as well as internal reviews. 

The EIA and JCS model documentation provides considerably more in¬ 
formation than the ERS format, since the latter is really a pointer system to 
help interested parties obtain more detailed information if desired. However, 
ERS also publishes reports on some of the major models. For example, a 
report on the ERS World Grain-Oil-Seeds-Livestock Model is sixty-four 
pages and includes a narrative description, illustrations of model equations 
and linkages, and values of key model parameters. 14 This report is backed up 
by an even longer technical report, also prepared by ERS staff. This suggests 
that even if modeling information available through directories or other 
public access mechanisms is limited, more detailed information may be 
available through technical reports prepared by agency (and/or consultant) 
staff and also by articles in the published literature. Even the EIA’s detailed 



Table 11.3 

Illustrative Agency Formats for Model Documentation 

Economic Research Service (Department of Agriculture)* 

• Model name 

• Responsible person(s) 

• Mode! description 

• Model applications 

• Operating and updating costs 

Joint Chiefs of Staff (Department of Defense) b 

• Model title 

• Model type 

• Proponent {who maintains model) 

• Developer 

• Purpose 

• General description 

• Date implemented 

• Input 

• Output 

• Model limitations 

• Hardware 

• Software 

• Time requirements 

• Security classification 

• Frequency of use 

• Users 

• Point of contact 

• Miscellaneous 

• Keyword listing 

Energy information Administration (Department of Energy) c 

• Model name 

• Acronym 

• Abstract 

• Status 

• Part of another model 

• Sponsoring agency, office, division, branch 

• Model contact 

• Documentation 

• Archive tapefsl and installation manuai(s) 

• Reviews conducted {of model) 

• Purpose 

• Energy system described by model 

• Coverage {e.g., geographic, time unit/frequency) 

• Special features 

• Modeling features 
—Model structure 
—Modeling technique 
—Model interfaces 
—Input data 

—Date sources 
—Output data 

• Computing environment 
—Language used 

—Core memory requirements 
—Estimated cost to run 
—Special features 

• Status of evaluation efforts 

• Date of last model update 

a See U.S. Department of Agriculture, Economics and Statistics Service, Agricultural and 
Other Economic Models of the Economics and Statistics Service (April 1981). According to 
USDA personnel, this document is stiil relatively current, and no update has been scheduled. 
& Joint Chiefs of Staff, Joint Analysis Directorate, “Memorandum for Agencies and Organiza¬ 
tions Involved in Wargaming and Military 7 Simulation Modeling,” re “Catalog of Wargaming 
and Military Simulation Models,” June 1, 1984. 
c US. Department of Energy; Energy 7 Information Administration, Directory of Energy Infor¬ 
mation Administration Model Abstracts, February 7 1, 1985. 
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public documentation is only an abstract of more extensive information 
available from knowledgeable EIA personnel. 

The agencies that use computer models to support major public informa¬ 
tion products (e.g., statistical reports on forecasts) generally have estab¬ 
lished means to make modeling information available. Other agencies have 
not explicitly dealt with the access question. Some simply recite the Freedom 
of Information Act. Others suggest that information would be made avaii- 
able if requested. There may not be a real issue here, except to the extent that 
modeling and decision support information is considered classified (primar¬ 
ily with respect to military applications) or subject to executive privilege. 
Public access to models developed by government contractors can also be a 
problem. The public availability of such information appears to need clarifi¬ 
cation. Also, the current central access mechanisms (e.g., the National Tech¬ 
nical Information Service and the National Energy Software Center) could 
be reviewed for adequacy and possible modification. 


Further Research on the Development and Use of Computer 

Modeling and Decision Support 

Judging from the apparent extensive use of computer models and the pos¬ 
itive tone of agency comments, computer models and decision support have 
a significant impact on agency decision making. For example, the Antitrust 
Division of the Department of Justice, and in particular the Economic Pol¬ 
icy Office, stated that “the data manipulation and sophisticated economic 
and statistical analyses now used in connection with almost all matters could 
not be performed without computers. While it is impossible to estimate sav¬ 
ings in staff time by using computer support, such savings are clearly large.” 
Nonetheless, the results of the OTA survey suggest that the actual use of 
models for decision making has received little systematic study by fed¬ 
eral agencies. Very few (about 4 percent) of the agencies using computer 
models report having conducted or sponsored such studies. About 7 percent 
of agencies using decision support report having conducted or sponsored 
studies. 

Of the few agencies that were able to provide concrete examples of studies, 
only the Federal Emergency Management Agency (FEMA) documented a 
clearly relevant study program (being carried out both in-house and with 
NBS assistance). It is likely that some study programs also exist in other 
agencies, especially in DOD components, but that the details or even the 
existence of such studies are unknown to headquarters personnel. The re¬ 
sponses of the army, navy, and air force headquarters noted the decentral¬ 
ized nature of agency operations, which makes it difficult, absent a major 
data collection effort, to be fully knowledgeable about prior or ongoing 
studies. On the other hand, neither the JCS nor the Program Analysis and 
Evaluation Office (in the Office of the Secretary of Defense) indicated any 
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such studies, although these two components make heavy use of computer 
models. It is possible that such studies may be classified, although no indi¬ 
cation to this effect was made to OTA by knowledgeable DOD personnel. 
JCS staff state that no such studies are conducted because the substantial 
value of computer modeling is clear and undisputed and, in any event, eval¬ 
uation studies would be difficult to do, given the multiple factors that affect 
JCS decisions. Computer model results are just one input among many. 

On the other hand, FEMA has made a major commitment to evaluate its 
computer models, many of which are intended to support planning for, and 
decision making under, emergency conditions. For example, in 1982, FEMA 
prepared a 130-page report of the FEMA Modeling Task Force that outlined 
a comprehensive plan for review and evaluation of FEMA modeling and an¬ 
alytical activities. 15 In 1984, reports were issued on various FEMA models, 
including the dynamic general equilibrium model designed to simulate eco¬ 
nomic conditions before and after an emergency, including nuclear attack, 
general wartime mobilization, and other severe economic disruptions, 16 and 
the damage assessment model designed to estimate the effects of a nuclear 
attack on various critical resources such as livestock, crops, housing, hospi¬ 
tals, and physicians. 17 

These and other models are then to be evaluated within a framework de¬ 
veloped by the Center for Applied Mathematics of NBS under contract to 
FEMA. The evaluation procedure is intended to, among other things, test 
the extent to which a model meets user requirements. NBS has identified a 
wide range of analytical techniques for model evaluation: 18 


• Descriptive analysis (e.g., motivation of model, theoretical underpinnings, model 

development). 

• Program verification and analysis (e.g., review’ of documentation and source code, 

model implementation). 

• Data audit (e.g., review of documentation, analysis of computerized files). 

• Sensitivity analysis (e.g., error analysis, statistical analysis, model stability). 

• Program usability (e.g., user-model interface, maintenance and update proce¬ 
dures). 


This NBS effort for FEMA represents a continuation of and builds on 
earlier work conducted in part for EIA and could very well serve as a proto¬ 
type for other agencies. 

Beyond this, there is a considerable body of research and discussion in the 
published academic and scholarly literature, 19 popular and trade press, and 
various research reports, for example, those sponsored by NSF on the use of 
models and decision analysis in risk assessment. 20 Also, variants of com- 
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puter modeling and decision analysis are being used in the development of 
computer-based expert systems and artificial intelligence. 

In sum, however, although many agencies believe in the utility of com¬ 
puter modeling and decision analytic techniques, few apparently think that 
studies are worth the time and resources. Nonetheless, it seems highly un¬ 
likely that all agencies are making the best and most cost-effective use of 
such techniques. A coordinated, modest research program could help illumi¬ 
nate the kinds of techniques and applications that are working well and 
those that are not. The results of such research would presumably facilitate 
the exchange of knowledge about computer modeling and decision support 
and lead to improved cost-effectiveness. The results would also be helpful to 
the development of model guidelines. 

In addition to encouraging and funding research, other mechanisms for 
sharing knowledge could be encouraged, such as professional forums for 
model developers and users (as has been tried in, for example, the energy 
and water resource modeling areas) and additional training opportunities. 

The limited research that is available, primarily academic research on 
model implementation, suggests that models (and, by extension, other deci¬ 
sion analytic techniques) can and do have a significant impact on agency 
decision making. Models may become a significant element in the process of 
negotiation over assumptions and options that is an integral part of agency 
(and, in general, political) decision making. However, models can be mis¬ 
used and abused. It may be important to understand the models and their 
roles in order to understand the ultimate decision. 21 

From this perspective, then, the results of further research may provide 
some new insights as to what kinds of questions should be asked and infor¬ 
mation requested in conducting oversight on agency decisions and what 
kinds of techniques might be useful in program evaluations and audits con¬ 
ducted by GAO and others. GAO and agency program evaluation and audit 
offices are generally very active and look for ways to improve evaluation and 
audit methodologies. Indeed, GAO is required, by the Congressional Bud¬ 
get Act of 1974, to monitor and recommend improvements in program and 
budget information for congressional use. GAO has identified needed im¬ 
provements in DOD’s planning, programming, and budgeting system, in the 
Environmental Protection Agency’s cost-benefit analyses of environmental 
regulations, and in DOD’s procedures for estimating weapons system 
costs. 22 In all these areas, decision analytic techniques have a potential role, 
especially techniques that combine quantitative and qualitative information, 
identify ranges of uncertainty, and specify the nature and extent of subjec¬ 
tive value judgments to the extent present in the analysis. GAO and other 
audit agencies could experiment with such decision analytic techniques to 
ascertain their potential to improve program and budget information for 
congressional use. 
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Further Testing and Development of the Decision 

Conference Technique 

Despite the widespread and frequently sophisticated use of computer- 
based decision support by federal agencies, the results of this effort appear 
to be used largely by agency staff or, at the most, presented to agency deci¬ 
sion makers for consideration along with other inputs. There appear to be 
relatively few situations in which the decision makers themselves actively 
participate in the decision analytic process. OTA located only one agency 
that has a formal program to do this: the decision conference facility of the 
Office of Program Planning and Evaluation in the Department of Com¬ 
merce (DOC). 

This DOC decision conference facility is used to bring key staff and deci¬ 
sion makers together for, typically, one or two days to work through a deci¬ 
sion problem using appropriate computer and analytical tools. Decision 
conference staff do advance work prior to the conference and serve as facili¬ 
tators, analytical experts, and rapporteurs during the conference. But the 
primary participants are the decision maker(s) and his or her staff. The 
DOC decision conferences use a wide range of computer-assisted analytical 
techniques, including spreadsheet software, quantitative, and qualitative 
judgmental, depending on what is most useful. 23 A list of illustrative deci¬ 
sion conferences is shown in Table 11.4. 

OTA found that DOD does not appear to have such a facility, despite 
DOD’s extensive use of computer-based decision analytic techniques. DOD 
does have numerous decision analytic support centers throughout the vari¬ 
ous service branches and commands, but they are at the staff and research 
levels. For example, the JCS staff conducts extensive studies (in-house and 
by contract) using modeling and decision analytic techniques, but the Joint 
Chiefs themselves do not normally participate, except to the extent of ap¬ 
proving the major studies. The results of selected decision analytic studies 
are presented to the Joint Chiefs when relevant to a decision problem at 
hand. 

The decision conference appears to have substantial potential, but the 
general consensus among practitioners is that further development and test¬ 
ing are needed prior to widespread application. Moreover, few decision 
makers are even aware of the technique, and even fewer have tried it. 

One of the keys to a successful decision conference is the direct and full 
participation of the decision makers. In order to have greater use of the tech¬ 
nique, decision makers need both greater awareness and greater understand¬ 
ing of the technique. Conducting pilot tests in selected programmatic areas, 
holding a workshop or conference, and commissioning a special report on 
the subject are actions that could help improve awareness and understand¬ 
ing. 

One of the areas thought to be most suited for the decision conference 
approach is R&D decision making. The National Marine Fisheries Service 
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Table 11.4 

Illustrative Decision Conferences Conducted by the Office of Program Planning 
and Evaluation, U.S. Department of Commerce, 1984-1985 

1. Development of Program and Budget Priorities for the U.S. 

National Marine Fisheries Service: (a) FY1986; (b) FY 1987. 

2. Promotion of Tourism to the United States—An Assess¬ 
ment of Alternative Marketing Strategies Available to 
the Department of Commerce in Six Regional Foreign 
Markets. 

3. Review of Alternative Programs and Service Delivery 
Strategies for the Minority Business Development Agency. 

4. Allocation of Saltonstall/Kennedy Fisheries Development 
Grant Program Funds—Priority Setting for Grant Appli¬ 
cations. 

5. Assessment of Alternative Foreign Trade Strategies for 
Promoting the Export of Auto Parts to Japan. 

6. Development and Evaluation of Alternative Staffing Stan¬ 
dards for Selected, Govemmentwide Administrative 
Functions (President’s Council on Management improve¬ 
ments): (a) Personnel; (b) Procurement; (c) Warehousing. 

7. Assessment of Alternative Long-Term Goals, Strategies 
and Implementation Mechanisms for the Telecommuni¬ 
cations, Computer, and Information Programs of the 
Department of Commerce. 

8. Assessment of Alternative Long-Term Strategies for 
Promoting Technological Innovation and the Transfer of 
Technology from Federal Laboratories to the Private Sec¬ 
tor (Preliminary). 

9. Assessment of Alternative Operating Objectives and 
Resource Allocations for Selected Administrative Activi¬ 
ties of the Department of Commerce: a) Personnel and 
Civil Rights Functions; b) Management and Information 
Systems Development; c) Financial Assistant Oversight 
Activities; and d) Regional Administrative Support Oper¬ 
ations. 

10. Alternative Programmatic Allocation of Field Personnel 
Resources, Center for Food Safety and Applied Nutrition 
of the U.S. Food and Drug Administration. 

Source: Office of Program Planning and Evaluation/Department of Commerce. 


(NMFS) (DOC) used a decision conference for decisions on the R&D budget 
for fiscal years 1986 and 1987. However, it should be noted that NMFS had 
been exploring decision analysis for several years, and thus appears to have 
been favorably predisposed. 24 Decision analytic studies also have been used 
as significant input to R&D decisions at DOC although not in the decision 
conference format adopted at DOC. At DOC, decision conferences have 
been conducted on budget, programmatic, and strategic decisions. 

The real power of the decision conference technique (or concept) is its 
potential to bring the full range of computer tools, models, analytical tech¬ 
niques, and the like into focus for the decision maker within a framework 
that is relevant to the decision maker. This is a concept that has been visual¬ 
ized and partially developed over the past twenty years or so by numerous 
researchers and innovators. 25 Table 11.5 places the decision conference in the 
context of other computer-supported conference room concepts. Different 
decision conference configurations are possible. For example, DOC uses 
software from the electronic boardroom and information center concepts in 
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addition to the software listed under decision conference, and it uses the 
orgware (i.e., organizational data and procedures) from the electronic 
boardroom and information center instead of the orgware listed under deci¬ 
sion conference. 

Overall, the decision conference concept is quite flexible, and many of the 
elements of the various concepts shown in Table 11.5 are interchangeable. 
Thus, it is feasible for a computer- or video-conferencing capability, for 
example, to be added to the decision conference. Indeed, OTA’s survey 
revealed that some agencies are already using computer conferencing, al¬ 
though not as part of decision conferences. For example, the U.S. Geologi¬ 
cal Survey makes extensive use of computer-conferencing on such diverse 
topics as cartography, geoscience, computer hardware and software prob¬ 
lems, news releases, and Mount Saint Helens’ volcanic activity bulletins. 

Another variation on the decision conference concept, interactive man¬ 
agement, is intended to deal with three principal functions of managers: 
intelligence (finding and clarifying problems), design (generating or concep¬ 
tualizing new or improved alternative solutions), and choice (selecting the 
preferred solution). 26 Like other decision conference concepts, the interac¬ 
tive management approach utilizes a situation room with appropriate audio¬ 
visual and computer support. What distinguishes interactive management is 
the explicit focus on intelligence, design, and choice and the use of a specific 
set of methodologies to structure ideas, design alternatives, and analyze 
trade-offs. 27 Several federal agencies have utilized the interactive manage¬ 
ment decision approach, including the Forest Service and Agricultural Re¬ 
search Service (Department of Agriculture), National Marine Fisheries 
Service (DOC), and Food and Drug Administration (Department of Health 
and Human Services). 28 

In sum, Kraemer and King’s 1983 prognosis that computer-supported 
conference techniques are “likely to grow at a slow pace over the next 2 
years, and pick up a bit thereafter” 29 may be coming true. It is now well over 
two years later, and the decision conference technique (sometimes also 
known under the rubric of group decision support systems or strategic plan¬ 
ning decision systems) is now considered to be at the cutting edge of com¬ 
puter-based decision analysis. 30 

NOTES 

Excerpted and edited from Fred B. Wood, “Computer Modeling, Decision 
Support, and Government Foresight,” in OTA, Federal Government Information 
Technology: Management , Security ; and Congressional Oversight , OX4-CIT-297 
(Washington, D.C., February 1986), pp. 103-136. The views expressed are those of 
the author and not necessarily those of the OTA, Technology Assessment Board, or 
U.S. Congress. 

1. See Saul I. Gass and Roger L. Sisson, A Guide to Models in Governmental 
Planning and Operations , report by Mathematics, Inc., prepared for U.S. Environ- 



254 


Computer-Aided Decision Analysis 


mental Protection Agency (August 1974); US. Congress, Office of Technology As¬ 
sessment, Information Technology R&D: Critical Trends and Issues , OTA-CIT-268 
(Washington, D.C.: US. Government Printing Office, February 1985). 
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1960s. The first global coupled atmosphere-ocean model was run in the mid-1970s 
on the state-of-the-art third-generation computers (e.g., IBM 360-195). Today, the 
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(e.g., Cray-2) and are providing the impetus for climate modelers to move up to even 
more powerful supercomputers. 

3. Stephen E. Frantzich, “Congressional Applications of Information Technol¬ 
ogy,” OTA contractor report prepared by Congressional Data Associates (February 
1985). 

4. Robert Miewald, Keith Mueller, and Robert Sittig, “State Legislature Use of 
Information Technology in Oversight,” OTA contractor report prepared by the Uni¬ 
versity of Nebraska-LIncoln (January 1985). 

5. US. General Accounting Office, Survey to Identify Models Used by Executive 
Agencies in the Policymaking Process, GAO/PAD-82-46 (September 24, 1982). 

6. Howard Raiffa, Decision Analysis (Reading, MA: Addison-Wesley, 1968). 
Also see Rex V. Brown, “A Brief Review of Executive Agency Uses of Personalized 
Decision Analysis and Support,” OTA contractor report prepared by Decision Sci¬ 
ence Consortium (March 1985). 

7. US. General Accounting Office, Ways to Improve Management of Federally 
Funded Computerized Models (August 23, 1976); Models and Their Role in GAO 
(October 1978); Guidelines for Model Evaluation (January 1979); U.S. Department 
of Commerce, National Bureau of Standards, Utility and Use of Large-Scale Mathe¬ 
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8. See, for example, Brian Wynne, “The Institutional Context of Science, 
Models, and Policy: The IIASA Energy Study,” Policy Sciences 17, no. 3 (November 
1984): 277-320; W. Hafele and H. H. Rogner, “A Technical Appraisal of the IIASA 
Energy Scenarios? A Rebuttal,” Policy Sciences 17, no. 4 (December 1984): 
341-365; Bill Keepin and Brian Wynne, “Technical Analysis of IIASA Energy Sce¬ 
narios,” Nature 312 (December 1984): 691-695; and David Dickson, “Global Energy 
Study Under Fire,” Science 221 (January 1985): 4. For a discussion of errors in fore¬ 
casting models, see William Ascher, Forecasting: An Appraisal for Policymakers and 
Planners (Baltimore, MD: Johns Hopkins University Press, 1978). 

9. See, for example, GAO, Guidelines; NBS, Utility; Richard Richels, “Building 
Good Models Is Not Enough,” Interfaces 11, no. 4 (August 1981): 48-51; and Saul I. 
Gass and Lambert S. Joel, “Concepts of Model Confidence,” Computers and Opera¬ 
tions Research 8, no. 4 (1981): 341-346. 

10. US. Congress, Office of Technology Assessment, Use of Models for Water 
Resources Management, Planning, and Policy (Washington, D.C.: U.S. Government 
Printing Office, August 1982). 

11. Public Law 93-275, sec. 57(B)(1) as amended by Public Law 94-385. 

12. See Energy Information Administration Order No. El5910.3A, “Guidelines 
and Procedures for Model and Analysis Documentation,” October 1, 1982, and Or- 
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Administration Model Abstracts (February 1, 1985); and Professional Audit Review 
Team, Performance Evaluation of the Energy Information Administration, report to 
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14. Karen Liu and Vernon O. Roninger, The World Grain-Oil-Seeds-Livestock 
(GOL) Model, A Simplified Version, ERS Staff Report No. AGE5850128, U.S. De¬ 
partment of Agriculture, Economic Research Service, International Economics Divi¬ 
sion (February 1985). 
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